What is the METR benchmark and what does it measure?

METR measures the time horizon and reliability of AI models completing tasks. It benchmarks models against human professionals and assesses task duration and success rate across different task complexities.

How has the task completion time for AI models improved over the years?

Task completion time with 50% success rate has significantly increased, from 2 seconds for GPT-2 in 2019 to around 59 minutes for Claude 3.7 Sonnet in 2025. This improvement is driven by better reasoning, code generation, and tool use.

How does the GDPval benchmark evaluate AI performance?

GDPval evaluates AI models against industry experts on economically valuable real-world tasks. It uses a win rate metric to compare AI output quality to that of professionals with over a decade of experience.

In which sectors are AI models currently performing closest to human experts?

AI models are nearing parity with human experts in sectors like government, retail, IT, and software development. They excel at tasks involving data synthesis, document formatting, and customer service, but still lag in complex, context-heavy roles.

Why do AI models struggle with research synthesis tasks?

AI models struggle with research synthesis due to difficulties in finding comprehensive and foundational sources, extracting key facts efficiently, and maintaining verifiability of citations. Experts possess contextual knowledge that models lack.

What are the key differences between METR, GDPval, and Deep Scholar Bench?

METR focuses on task duration and 50% reliability, GDPval on economic value via win rates against experts, and Deep Scholar Bench on the quality of research synthesis. They highlight different aspects of AI capability and its limitations.

What are the main limitations of current AI models?

Current models have low confidence in tasks requiring extensive context, struggle with adversarial environments, lack high reliability (95%), and show limited generalization beyond software and knowledge work. Referencing knowledge bases and ensuring verifiability remain significant gaps.

What is stopping AI from tackling long-tail problems?

The challenge lies in a mix of data limitations and fundamental model capabilities. Creating representative tasks and closing the gap between model capabilities and task requirements, especially in specialized domains like robotics, remains difficult.

Key Moments

Stanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks

Q: What are common failure modes for AI agents in complex tasks?

Common failure modes include poor planning, incorrect reasoning, premature task abandonment, repetitive loops, and poor tool choice. Analyzing these failures is crucial for model improvement.

Q: What is the Deep Scholar Bench and what does it evaluate?

Deep Scholar Bench is a Stanford benchmark that measures AI's ability in deep research synthesis, specifically generating related work sections for academic papers. It evaluates knowledge synthesis, retrieval quality, and verifiability of citations.

Stanford Online

Education6 min read76 min video

Aug 3, 2026|60 views|2

Stanford Stanford Online

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

AI agents can now tackle tasks lasting hours, but achieving high reliability (80% success) remains challenging, with performance varying significantly by task type and domain.

Key Insights

The METR benchmark shows that AI models have doubled their task completion time horizon (at 50% success rate) approximately every 7 months, reaching durations of a few hours for advanced models.

GDP-Val reveals that for economically valuable tasks, AI models only achieve a win rate of around 47.6% against industry professionals, indicating significant gaps in real-world applicability.

Deep Scholar Bench shows that current AI systems struggle with knowledge synthesis and retrieval for research tasks, with none exceeding 19% performance on key metrics like coherence, relevance, and verifiability.

While AI can complete longer tasks, reliability is a major issue; achieving an 80% success rate limits task duration to minutes, not hours, highlighting a significant gap between capability and practical deployment.

Failure modes in AI agents include poor planning, incorrect reasoning, premature task abandonment, repetitive loops, and an inability to consult or correctly use reference data.

AI performance is strongest in well-specified tasks within domains like software engineering and ML research, but struggles with ambiguous prompts, low-context scenarios, and tasks requiring deep domain expertise or novel information retrieval.

Measuring AI's evolving task duration capabilities

The lecture introduces the METR (Measuring AI Time Horizon) benchmark, designed to quantify the duration of tasks AI agents can complete. Historically, AI models struggled with multi-turn conversations, but recent models maintain context over much longer interactions. METR categorizes tasks into small (1-30 seconds), edgecast (1 minute to 30 hours), and rebench (up to 8 hours), totaling 170 tasks. The benchmark correlates human completion time with AI success rates. Trends show a significant increase in the time horizon for AI task completion, with models doubling their 50% success rate duration roughly every 7 months. For instance, GPT-2 in 2019 handled tasks around 2 seconds, while models like GPT-4 in 2023 can manage tasks of several minutes, and models like Claude 3.7 Sonnet in 2025 can reach up to 59 minutes at 50% success. This progress is attributed to improvements in logical reasoning, code generation, tool use, reliability, error recovery, and goal awareness. However, it's crucial to note that these durations are often at a 50% success rate, meaning the task might not be completed successfully. This highlights the difference between capability and reliability for real-world deployment.

Assessing AI's economic value in real-world tasks

The GDP-Val benchmark shifts focus to evaluating AI performance on economically valuable tasks, comparing model output directly against industry experts with over a decade of experience. This benchmark covers nine sectors, including healthcare, finance, manufacturing, and creative industries, with 320 tasks sourced from professionals. These tasks can be text-based or multimodal, with an average value of $400 in the gold subset. The evaluation metric is the win rate against human experts. Unlike the exponential trend observed in METR, GDP-Val shows a more linear improvement. For example, GPT-40 in 2024 had a win rate around 12.4%, while Claude Opus 4.1 in 2024 reached 47.6%. This benchmark reveals that while AI is improving, it still falls short of human experts in many real-world, economically significant tasks. This implies that current AI assistance in professional workflows is most effective when coupled with human oversight, where humans architect the problems and AI executes specific steps. The value of AI is amplified when it complements human expertise rather than replacing it entirely.

The challenge of deep research and synthesis

Deep Scholar Bench targets AI's ability to perform deep research synthesis, specifically generating the related work section for academic papers. This is a critical and challenging task requiring not just information retrieval but also synthesis, coherence, and verifiability of citations. The benchmark uses recent PhD-level papers across 22 domains, updated monthly, to avoid data contamination. Evaluation axes include knowledge synthesis (coherence, capturing key facts), retrieval quality (relevance of sources, document importance), and verifiability (claims backed by citations). A key finding is that none of the current systems exceed 19% performance on these metrics, indicating substantial room for improvement. Models often miss key facts, struggle to find comprehensive or foundational sources, and have difficulty verifying claims. Even when provided with perfect sources, AI extracts key facts with only about 50% coverage. This benchmark underscores that tasks requiring deep knowledge integration, contextual understanding, and reliable synthesis of information are still major hurdles for AI, even when compared to an expert who has internalized years of domain knowledge.

Reliability gaps: 50% success vs. 80% success

While METR highlights progress in task duration, the crucial metric of reliability is also examined. The data shows a significant divergence between achieving 50% success rate and higher reliability targets like 80%. For instance, a model might achieve a 59-minute task completion at 50% success, but at 80% success, that duration drops dramatically to around 8-10 minutes. This substantial gap indicates that even if models can technically perform tasks for longer durations, their output is not consistently reliable enough for critical applications. This low reliability is a major obstacle for deploying AI in real-world scenarios where consistent performance is essential.

Common AI failure modes

The analysis of AI failures across benchmarks like GDP-Val and others reveals recurring issues. These include poor planning (inability to break down tasks), incorrect mental math or reasoning, premature task abandonment, repetitive loops, and poor tool choice. Particularly problematic is the failure to correctly consult or utilize reference data. Models may promise to consult data but instead hallucinate or ignore it. Formatting errors also persist. While some models like GPT-5 show fewer instruction-following errors, the overall pattern suggests that AI agents still require robust error handling and improved reasoning to be dependable.

Context dependency and the contractor analogy

A significant challenge for AI agents is their reliance on context. In real-world tasks, human experts leverage extensive domain knowledge and contextual understanding, often accumulated over years. When AI agents lack this explicit context in their prompts, their performance degrades substantially. Benchmarks suggest AI performance is closer to that of an external contractor with no prior knowledge of a codebase than a repository maintainer who possesses deep context. This highlights that while AI can execute well-defined instructions, comprehending and integrating implicit context, a hallmark of human expertise, remains a difficult area.

Variations in performance across domains and modalities

AI performance is not uniform across all task types. Domains like software engineering and ML research, where tasks are often more digital and well-specified, have seen significant progress. Models demonstrate strong capabilities in tasks such as code generation, data manipulation, and basic research synthesis. However, performance declines with longer, more complex, or more subjective tasks. Multimodal capabilities are improving, but some models excel more at text-based tasks while others handle visuals, audio, and spreadsheets better. The effectiveness of AI also varies by profession, with some roles like counter clerks or software developers seeing AI performance approach or even match human averages, while others remain far behind.

The future outlook: AI as an assistant, not a replacement

While AI capabilities are rapidly advancing, the current trajectory suggests AI will largely function as an assistant rather than a full replacement for human experts in the near future. The focus is on AI co-piloting and augmentation, especially in complex fields requiring deep contextual understanding, nuanced judgment, and a high degree of reliability. The "long tail" of challenging tasks, distributed systems, and highly specialized research continues to be difficult for AI. Future progress will likely depend on continued improvements in fundamental model capabilities, better data collection strategies for specialized domains, and more sophisticated evaluation metrics that capture reliability, context integration, and true economic value.

Mentioned in This Episode

●Software & Apps

●Companies

●Studies Cited

Common Questions

Evaluating AI agents is challenging due to the complexity and length of tasks. Traditional benchmarks are saturating, necessitating new metrics for capability, economic value, and long-term task completion.

Topics

AI & Machine Learning Technology & Innovation Long-horizon Tasks Economic Impact Of AI AI Benchmarks Agentic Evaluation Model Reliability Research Synthesis AI Failure Modes

Mentioned in this video

Software & Apps

GPT-4

A more recent model compared to GPT-2, capable of handling tasks for a few minutes with 50% success rate.

GPT-4o

A model evaluated in the GDPval benchmark, showing a win rate around 12.4% against industry professionals in 2024.

GPT-5

Mentioned as a model with improved capabilities, particularly in instruction following, and showing better performance in the GDPval benchmark.

Perplexity

A platform with a deep research feature, mentioned in the context of the Deep Scholar Bench.

GPT-2

Mentioned as an older model from 2019, which could only handle tasks around 2 seconds long with 50% success rate.

Studies & Research

Deep Scholar Bench

A benchmark developed at Stanford to measure AI's ability to perform deep research synthesis, specifically generating related work sections for academic papers.

Companies

OpenAI

Mentioned as the developer of the GDPval benchmark, which evaluates model performance on real-world economically valuable tasks.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free