Key Moments
Why The Harness Matters More Than The Model | YC Paper Club
Y CombinatorWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Harnesses, often dismissed as mere prompt engineering, are crucial for AI performance, turning 30% scores into 95% success rates and enabling breakthroughs like Arcade AI.
Key Insights
A significant improvement, from 30% to 95%, was achieved on ARC-AGI benchmarks by simply employing a better 'harness' with the same model weights.
The evolution of AI agents has moved from static harnesses to self-improving harnesses, marking a significant shift in their capabilities and adaptability.
Early AI agent development, like that of GPT-2, relied on very basic harnesses consisting of simple iteration loops and sampling methods.
The development of harnesses has progressed through stages including few-shot learning, chain-of-thought prompting, tool use (Toolformer, WebGPT), memory (MemGPT), skills, and multi-agent systems (Voyager, Reflect).
Open Jarvis aims to build the core of personal AI on-device, offering privacy and cost savings compared to cloud-based solutions, with models lagging only 6-12 months behind state-of-the-art.
YC's QM framework provides each employee with a customizable, open-source AI assistant, managing individual contexts and enabling collaborative tasks through Slack integrations.
Harnesses: More Than Just Scaffolding
The prevailing view often dismisses 'harnesses' as mere scaffolding or prompt engineering, not worthy of serious research. However, this perspective drastically underestimates their impact. The presentation highlights a stark example where the same model weights achieved a mere 30% on the ARC-AGI benchmark, but with a superior harness, this performance jumped to an impressive 95%. This dramatic improvement underscores that the effectiveness of an AI agent is not solely dependent on the model's weights but significantly on the framework, or 'harness', that guides its interaction with the world and its tasks. The success of companies like Arcade AI is directly attributed to the difference between their first and second harnesses, proving that harness development is a critical research area, not just an implementation detail.
The evolution of AI agent frameworks
The journey of AI agent frameworks has been a continuous progression. Early systems, exemplified by GPT-2's initial harness in February 2019, were rudimentary, primarily consisting of a loop that continued until the sequence end, coupled with 'top P sampling'. This basic structure lacked tool calls or complex skills. Over time, harnesses evolved to incorporate few-shot learning through in-context examples, aiming to guide the model with similar instances. This was followed by the 'chain of thought' prompting, which breaks down complex reasoning into smaller, manageable steps across more tokens. The introduction of tool use, as seen in Toolformer and WebGPT, allowed agents to interact with external functionalities, like calling a Python function for calculations. MemGPT then introduced persistent memory, enabling agents to read and write to their own context. Voyager further expanded this by enabling agents to chain tools together to accomplish tasks and learn from them, leading to the concept of 'skills'. This evolution demonstrates a consistent effort to enhance the agent's capabilities and autonomy.
The shift to self-improving and multi-agent systems
The past six months have been particularly transformative, shifting the focus from static harnesses to self-improving ones. This new era is characterized by architectures that allow agents to reflect on their own outputs and refine them, or to collaborate with other agents. Systems like React and Self-Refine exemplify this, where multiple agents with different roles can work together to improve the context. Voyager's approach to chaining tools and distilling them back into the system prompt for perpetual learning introduced the concept of 'skills,' applied effectively in environments like Minecraft. InterCode further blurred the lines by treating code generation as an immediate skill or tool. The idea of multi-agent systems extends to creating sub-agents that can operate and interact, managed by a coordinating agent, forming what is termed 'Harness V1'. This static structure, while powerful, still relies on external infrastructure like G-stack for execution. The true innovation lies in allowing the harness itself to learn and adapt.
DSPy and Darwin Godel: Harnesses learning to improve
The cutting edge of harness development involves frameworks that enable the harness to learn and evolve. DSPy (Declare, Specify, Predict) is a prime example, using a small set of training examples to learn optimal system prompts through techniques like genetic programming, rather than traditional backpropagation. This provides full control over the system prompt. Darwin Godel takes this a step further by allowing not only the system prompt but also the harness code itself to be modified. This creates a meta-level system where an archive of harnesses and prompts is maintained, experimented with, and iteratively improved based on a fitness function. The ultimate goal is a meta-harness that produces other harnesses, representing a deeply recursive and powerful approach to AI development. This ongoing evolution aims to create agents that can continuously refine their own underlying code and prompts.
Open Jarvis: On-device personal AI
The Open Jarvis project tackles the growing need for personal AI that is private, cost-effective, and energy-efficient. Unlike current cloud-dependent systems, Open Jarvis aims to run all core AI functionalities—model inference, agent execution, memory, and learning—entirely on-device. This approach is made feasible by increasingly capable local language models that now lag only 6-12 months behind state-of-the-art cloud models. The project proposes a modular architecture comprising user interfaces, agent logic, various LLM engines (like Qwen, GPT-OSS, Gemma 3N), inference engines (Olama, LLaMA CPP), tools, and learning mechanisms. By optimizing these components, Open Jarvis seeks to provide a competitive alternative to cloud-based AI, enabling users to retain full control over their data and reduce operational costs significantly. The project emphasizes a plug-and-play approach, allowing users to integrate their personal AI into existing workflows and experience the benefits of local, private AI.
YC's QM: An AI assistant for every employee
YC's QM (Quantum Mechanics, presumably) framework provides every employee with a customizable, open-source AI assistant accessible via Slack or a web interface. Each QM agent operates within its user's personal context, complete with its own test environment, files, and schedule, while also supporting collaborative group settings. Users leverage QM for a wide range of tasks, including email sorting, legal and financial workflows, document editing, data extraction, and even running internal web applications. The development of QM stems from years of internal agent projects, evolving from a simple 'General Agent' in January 2025 to more sophisticated versions that integrated with Slack, gained tools, and could execute code. The goal is to provide a powerful, yet manageable, AI assistant that enhances employee productivity across diverse daily tasks, overcoming the challenges of managing large fleets of individual agent instances seen in earlier iterations like Hermes.
The importance of harness expressiveness and continuous improvement
The core argument presented is that the expressiveness of a harness is paramount, directly impacting an AI agent's capabilities. A more expressive harness, which allows agents to access Python environments, create sub-agents programmatically, utilize feedback mechanisms, and interact with external memory, unlocks functionalities that the raw language model alone cannot achieve. This is akin to upgrading from a Turing machine to a Von Neumann architecture, enabling complex read/write operations on external memory. Furthermore, the evolution towards 'continual harnesses' and agent operating systems signifies a paradigm shift. These systems allow for continuous learning, memory management (including garbage collection and pruning), and dynamic adaptation of the harness itself. The ability for an agent to manage its own sub-agents, select its execution environment, and learn from past experiences—even across long-running tasks lasting days—is critical for building robust and adaptable AI systems. This continuous improvement cycle, driven by both human oversight and autonomous refinement, is seen as the future of AI agent development.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
●People Referenced
Common Questions
Agent harnesses are essentially the scaffolding or command engineering that surrounds large language models. They are crucial because they enable AI systems to achieve significant performance improvements, with the difference between early and later harnesses showing an 18% jump in effectiveness, impacting the success of AI ventures.
Topics
Mentioned in this video
Mentioned as an AI model that initially resisted drawing a lobster but was eventually made to do so.
Supervised Fine-Tuning, a training procedure used after LoRA.
The initial version of the GPT model, used as an example for the earliest form of a test harness in February 2019.
A cloud-based language model used to optimize local language model configurations for Open Jarvis.
Used as a benchmark and example of early AI coding assistance.
An AI model whose recent results are being analyzed for out-of-loop experiments.
A language model that can be used as the engine for Open Jarvis.
Standard protocol for running tools and memory within Open Jarvis.
A database system used by QM to centralize agent conversations and context.
An enhanced interactive Python shell used as the environment for agents in Prime Agent.
A local language model that achieves performance comparable to Claude 4.6 Opus from August 2025.
A language model that can be used as the engine for Open Jarvis.
A language model family that can be used to optimize Open Jarvis configurations.
An agent that proved very expensive to run, costing $5000 without significant performance gains, highlighting cost-effectiveness.
A language model family that can be used to optimize Open Jarvis configurations.
An AI agent developed by NVIDIA, mentioned alongside AVO as recently released.
An inference engine for running language models.
One of the first models verified on a private dataset, achieving 30% accuracy initially, but improved to 95% with harnesses.
A system that allows AI models to output code, blurring the line between tools and skills.
A framework for optimizing the system prompt, using techniques like genetic programming instead of backpropagation.
An AI model that achieved 25.7% on a benchmark when tested with Prime Agent.
A benchmark for recreating full computer system emulators, such as the Gameboy Color.
A language model that can be used as the engine for Open Jarvis.
An AI model that can use tools, discussed as a step in the evolution of agent capabilities.
A research paper by Alex discussing recursive language models and sub-agent sessions.
Weight-based learning technique within Open Jarvis.
Mentioned as an example where agents might face rejection for certain types of queries (e.g., AI research, cybersecurity).
Graphics Processing Units, mentioned in the context of testing agent performance on GPU kernels.
An AI model whose recent results are being analyzed for out-of-loop experiments.
A project aiming to build the core of personal AI packages entirely on-device, exploring local execution, agent implementation, memory, and learning.
A cloud-based language model used to optimize local language model configurations for Open Jarvis.
Highlighted for its speed in adapting to new problems and distributions, and its performance in such scenarios.
A tool used for accessing the agent's dashboard from anywhere and receiving updates.
An early AI model that could interact with the web, discussed in the context of tool usage and evolution.
An inference engine for running language models.
An AI system that allows for reading and writing to its own context, enabling more dynamic memory management.
A foundational technology that enables advanced AI capabilities, thanked by the speaker for its role.
An AI model, likely referring to Claude Opus, that achieved 95.5% accuracy with Prime Agent.
A previous state-of-the-art model from August 2025, now comparable to newer local models like Qwen 3.8 27B.
An inference engine for running language models.
An AI concept where agents reflect on their outputs and improve them, discussed in the context of multi-agent systems.
Command-based learning technique within Open Jarvis.
A concept in AI agent development, discussed in the historical context of agent improvement.
A concept that extends prompt optimization to the harness code itself, allowing for self-modification of agents.
A concept related to multi-agent systems and self-reflection for context improvement.
A concept in AI agent development, discussed as part of the historical literature leading to current advancements.
Autoregressive Large Language Models, contrasted with Diffusion LLMs in terms of performance and computational density.
A reasoning technique in AI, contrasted with simpler output formats in early agent prompts.
Discussed in comparison to AR LLMs, noting that while they don't inherently outperform, a strategy of aggregation and partitioning can yield better results due to computational density.
A sampling strategy used in the early GPT-2 test harness.
An experiment involving automated research on nanoGPT, scaled up with significant computing resources.
A long-range evaluation task, similar to programming simulations, used to test agent performance.
A benchmark used for evaluating the performance of AI agents, particularly in mathematical reasoning tasks.
A paper from July 2020 that discussed the concept of learning from a few examples provided within the context.
An example task format used to illustrate the basic structure of early AI agent prompts and evaluation.
An AI model whose recent results are being analyzed for out-of-loop experiments.
Mentioned as a source for research papers and code repositories that an agent might explore.
An organization where Seth is a researcher and an author of Prime Agent.
An agent type running in virtual machines, used in a fleet at YC before the development of QM.
Mentioned for solving complex math problems with technology similar to QM's persistence tools.
A platform where a post questioned the validity of prompt engineering as a research topic for AI conferences.
A company whose success or failure is tied to the effectiveness of agent harnesses, highlighted by Seth.
Mentioned in relation to a CEO's presentation about measuring perplexity and machine intelligence.
Achieved 100% accuracy on a task where Claude Opus initially struggled, indicating the power of advanced AI tools.
Low-Rank Adaptation, a technique used in AI model training after ICL saturation.
Mentioned for solving complex math problems with technology similar to QM's persistence tools.
Mentioned for its new Mac Mini and its renewed focus on building custom accelerators for personal AI use cases.
An earlier agent system used at YC that provided each partner with their own customizable computing environment.
A research agent tasked by Chris Ray to perform specific tasks.
A presenter at the event, co-author of Open Jarvis, with a PhD supervised by Chris and Azalea.
Promoted to Head of YC Labs, presenting on QM.
A 'Principal Investigator' (PI) agent who assigns tasks to a research agent.
Seth's advisor, mentioned in relation to Seth's background as a researcher.
High-performance computing units used in clusters for running AI research and development tasks.
Hardware platform for running AI inference engines.
An AI agent system, discussed as part of the historical development of agent harnesses and skills.
An example of a system emulator that can be recreated and tested using the Emulator Bench.
More from Y Combinator
View all 631 summaries
58 minOpen Models Are Collapsing The Cost Of AI
22 minPaul Graham On Startups, Ambition, and Great Founders
60 minMax Junestrand: You Need The Willingness To Learn Faster Than Anyone Else
53 minGoing In Deep On Data | YC Paper Club
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free