Key Moments
Terminal-Bench: Evaluating AI Agents | Stanford CS329Z Guest Lecture with Alex Shaw & Ryan Marten
Key Moments
AI agents can now perform complex tasks in a terminal environment, but evaluating their performance is challenging due to the shift from deterministic code to probabilistic models, requiring new frameworks like Terminal-Bench and Harbor.
Key Insights
The evolution of software development has shifted from deductive, rule-based programming to inductive, machine learning-based approaches, with AI agents representing the latest frontier.
Terminal-Bench, co-created by Alex Shaw and Ryan Marten, serves as a benchmark for evaluating AI agents in a terminal environment, with over 171 contributors and 7 million runs.
Harbor is a framework designed to build agent evaluation benchmarks, abstracting infrastructure and promoting best practices for creating reliable, reproducible, and efficient evaluations.
Evaluating AI agents requires a new paradigm where outcomes, not code, are the source of truth, necessitating rigorous testing and validation to ensure reliability and prevent reward hacking.
The creation of effective agent tasks is challenging, involving the alignment of instructions with verifiers, prevention of reward manipulation, and ensuring realism and fairness in the evaluation.
The evolution from deductive to inductive software development
The lecture begins by tracing the historical trajectory of software development, starting from the 1940s with low-level machine code, progressing to assembly languages, and then to high-level languages like FORTRAN. Early programming was characterized by a deductive approach, where humans defined explicit rules and instructions. This shifted significantly with the rise of machine learning in the 2000s, embracing an inductive approach where models learn patterns from data. Chris Anderson's 2008 article highlighted this shift, suggesting that "the age of theory is over" and data abundance makes the scientific method obsolete. In this inductive paradigm, humans are responsible for data, labeling, and algorithm selection, while the training process derives the rules. This evolution has culminated in large language models (LLMs) and, more recently, AI agents, which can generate text that is translated into actions within a computing environment. This marks the peak of inductive software engineering, where AI agents perform tasks on computers without explicit human programming for every action. The core idea is that all digital technology development is becoming increasingly inductive, fundamentally changing the role of humans in software creation. This also implies that understanding software will shift from examining the code itself to analyzing its outputs and behavior, treating it as a 'black box'.
Terminal-Bench: A benchmark for agent capabilities
Terminal-Bench, developed by Alex Shaw and Ryan Marten, addresses the need for robust evaluation of AI agents. It positions the terminal as a primary interface for agents, given its text-based nature and ability to execute any command, making it uniquely suitable for LLMs. The benchmark has seen significant community involvement, with over 171 contributors, and has been run more than 7 million times, demonstrating its utility for both final model releases and intermediate training checkpoints. The inspiration for Terminal-Bench stemmed from prior work on the K Prize competition, which used a continuously updated version of SweBench. This experience highlighted the limitations of existing benchmarks and the desire to generalize the concept of computer tasks beyond just GitHub repositories and pull requests. The goal was to create a more comprehensive and flexible evaluation framework that could capture a wider range of agent capabilities. The project emphasizes the terminal's power as a universal interface, capable of replacing many other tools with a single command, and LLMs' proficiency in generating these commands.
Harbor: A framework for building and running evaluations
Harbor is the accompanying framework to Terminal-Bench, providing the tools and infrastructure for building agent evaluations. It aims to encapsulate the lessons learned from Terminal-Bench's development, enabling others to create their own benchmarks without rebuilding the underlying infrastructure. At its core, Harbor is designed for "evaluation-driven development," with the goal of understanding and improving agents. In the current landscape of probabilistic agents, evaluations and observations serve as the source of truth, rather than deterministic code. Harbor provides a standardized way to define tasks, environments, and verifiers, making the evaluation process more systematic. The framework is built to be reliable, reproducible, and efficient, allowing for parallel execution of tasks and providing detailed logs for analysis. It emphasizes modularity, allowing tasks and environments to be easily swapped, and supports various types of evaluations, including offline, automated assessments and potentially online, production-level monitoring.
Defining and creating agent tasks
Creating effective agent tasks is a complex undertaking. The key components of a Terminal-Bench task include instructions (the prompt), the environment (often a Dockerfile simulating a virtual computer), and verifiers (the logic to evaluate performance). The hardest aspects are aligning instructions with the verifier, preventing reward hacking, and ensuring the task is realistic, fair, and interesting. Instructions should be clear and concise, minimizing ambiguity. The environment should accurately mimic production settings, potentially including real-world software and services, while balancing openness (like internet access) against stability and safety concerns. For instance, a task involving reverse-engineering a label required adding distracting assets to make it more challenging. Verifiers are crucial for determining success, typically acting as binary classifiers (success/failure) but can also be continuous. They must precisely match the instructions, and issues like false positives/negatives and reward hacking must be actively mitigated. This often involves careful prompt engineering and robust verification logic, sometimes incorporating LLM-based judgments, although this introduces its own layer of complexity.
The challenge of reward hacking and verifier design
Reward hacking, where agents exploit loopholes in the evaluation system to gain rewards without achieving the intended outcome, is a significant challenge. This is an ongoing 'arms race' as more capable agents discover new vulnerabilities. Preventing it requires careful task design and robust verification. Harbor offers infrastructure-level solutions, such as running the agent and verifier in separate containers, to minimize interference. Task creators must anticipate potential reward hacks specific to their tasks. For example, exploiting software bugs in the Linux kernel could bypass certain verification checks. The design of the verifier is paramount; it must accurately reflect the task's objectives. While code-based checks are reliable, subjective tasks may require LLM-based verifiers, which introduce their own complexities and potential for randomness. The overall goal is to ensure that if an agent follows the instructions precisely, it should receive full credit, and vice versa. This balance is delicate and requires continuous refinement.
The role of infrastructure and maintenance
The underlying evaluation infrastructure, whether custom-built or using a framework like Harbor, must be reliable, reproducible, fast, and portable. Reproducibility is key given the probabilistic nature of agents, meaning the infrastructure itself should not introduce additional variability. Parallel processing is essential for speed, but managing shared resources requires careful consideration. Portability ensures that evaluations can be run consistently across different environments, from local development to large-scale deployment. This infrastructure often mirrors the challenges found in training and production environments, highlighting the interconnectedness of the AI development lifecycle. Maintenance is also critical, as tasks are complex software artifacts prone to bugs. This includes rigorous quality assurance, identifying and fixing negative patterns, and adapting to new agent capabilities, such as novel reward hacking techniques or improved solutions that might not be correctly evaluated by existing verifiers. Versioning of tasks and verifiers is essential for managing changes and ensuring comparable results over time.
Future directions and community involvement
The current format of Terminal-Bench, while valuable, is a simplified representation of real-world computer tasks. Future iterations are expected to increase environmental complexity, task horizon length, and coverage of diverse computational activities. The creators emphasize that building such benchmarks is not a scalable process for a small team; hence, the development of Harbor to empower the community to create and refine benchmarks tailored to specific domains and interests. This collaborative approach is crucial for advancing the field. The ultimate goal is to enable agents to tackle increasingly complex, real-world problems, moving beyond simple, deterministic tasks to more nuanced and open-ended challenges. The lecture concludes by acknowledging that while automated, offline evaluation is critical, other forms like online evaluation (monitoring agents in production) and manual evaluation also play roles, with Harbor and Terminal-Bench providing robust tools for the offline automated assessment.
Mentioned in This Episode
●Software & Apps
●Companies
●People Referenced
Best Practices for Agent Evaluation
Practical takeaways from this episode
Do This
Avoid This
Common Questions
Terminal Bench is a benchmark for evaluating AI agents, specifically designed to test their ability to solve complex, long-horizon tasks within a terminal environment. It was created to address the shift from purely deductive software engineering to more inductive and agent-based approaches in AI development.
Topics
Mentioned in this video
Co-creator of Terminal Bench and Harbor, presenting on the practical aspects of agent evaluation, the terminal's role, and the Harbor framework.
Inventor of Quicksort and the null pointer reference, quoted on the null pointer being his 'billion-dollar mistake'.
Author of a 2008 article suggesting the era of theory is over and data floods make the scientific method obsolete, influencing the shift towards inductive machine learning.
Pioneer in information theory, after whom Anthropic's Claude is named; his dream of building a thinking, learning machine is seen as realized with modern AI.
Co-creator of Terminal Bench and Harbor, discussing the evolution of software development and the shift towards inductive and agent-based approaches.
Credited with creating Fortran and quoted on the perception of early programmers as a 'priestly class'.
Creator of Keras and the ARC prize, quoted on agent programming being a form of machine learning and the need to treat generated code as a black box.
Part of the team that worked on Terminal Bench, mentioned as an amazing individual contributing to the project.
A framework and toolkit for building agent evaluations and benchmarks, derived from lessons learned in creating Terminal Bench.
Mentioned as a company that utilizes large amounts of data to find correlations without necessarily understanding causality, exemplifying the inductive approach in machine learning.
Mentioned in the context of software development and as an example of a platform where code is managed; the slogan 'every company is a software company' is attributed to it.
Cited as an example of a company using recommendation systems based on inductive machine learning approaches.
Cited as an example of a company using recommendation systems based on inductive machine learning approaches.
Mentioned for their blog post 'Demystifying Evaluations for AI Agents,' which heavily influenced the Harbor framework's design.
A hypothetical future version of GPT, mentioned as an example of how models are used in modern software development for tasks like phone number extraction.
The website where Terminal Bench results are published, offering detailed information beyond a simple score.
Containerization platform used to define and run environments for agent evaluation in Harbor and Terminal Bench.
A tool used to define and run multi-container Docker applications, useful for creating complex agent evaluation environments.
A hypothetical future version of GPT, mentioned as an example of how models are used in modern software development for tasks like phone number extraction.
A deep learning API, mentioned as being created by François Chollet.
A file used to define a Docker image and environment for running agent tasks.
AI model from Anthropic, mentioned in the context of its usage and evaluation.
The dominant tool for interacting with computers, highlighted as a powerful text-based interface that AI language models are uniquely suited to generate commands for.
Mentioned in the context of kernel bugs that could be exploited to bypass tasks in agent evaluations.
Mentioned as the product that popularized large language models, making them commercially successful and widely adopted.
Amazon Elastic Compute Cloud instance, used as an example to contrast the ease of setting up via a web interface versus a single command in the terminal.
One of the earliest high-level programming languages, mentioned as an example of early advancements in software engineering.
Mentioned as a release that occurred shortly before the development of Terminal Bench.
A benchmark for evaluating AI agents, particularly in software engineering tasks, which served as a foundation for the KPrize and influenced Terminal Bench.
A benchmark that is considered a subset of Terminal Bench, focusing more heavily on software engineering tasks.
A specific model version mentioned as an example of an agent performing a cloud coding task.
Mentioned in the context of agent interaction, specifically how instructions can be passed to a user agent that then guides the agent being evaluated.
More from Stanford Online
View all 186 summaries
49 minAI Costs: Tokens, Energy, Model Choice & API Keys | Stanford CS193T Lecture 7 with Cynthia Bailey
81 minOptimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan
70 minVibe Coding, AI Risk & AI Evaluation | Stanford CS193T Lecture 6 with Cynthia Bailey
59 minHow AI Agents Work: Connectors, MCP & Agent Safety | Stanford CS193T Lecture 5 with Cynthia Bailey
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free