Key Moments
⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev
Latent Space PodcastKey Moments
Software factories can optimize AI code generation, reducing PR costs from $80 to $15-20, but require significant engineering investment for companies with 5-10+ engineers.
Key Insights
Warp reduced its internal cost per Pull Request (PR) from approximately $80 to $15-20 by tuning models and optimizing sub-agent usage.
A software factory is defined as a generative system in the cloud where agents are assigned roles for software creation with a degree of autonomy, aiming for measurability and a closed-loop system.
Defining a software factory requires explicit definition through code (e.g., YAML) specifying repositories, MCPs, integrations, and workflows, treating it as a concrete, orchestratable, and measurable entity.
Centralizing agent conversations and traces provides real-time insights into a team's output, allowing for metrics like human intervention per PR, throughput, and cost analysis.
The core of a software factory's improvement loop lies in its measurability and observation, enabling 'scorers' or evaluators to assess agent performance on metrics like talkativeness, effectiveness, and code quality.
Benchmarking involves running coding agent tasks against different configurations of models and harnesses, providing company-specific results on cost vs. accuracy to inform routing strategies.
Defining the software factory as a code-driven, measurable system
A software factory is conceptualized not as a vague collection of automated tasks, but as a precisely defined, cloud-based generative system. At its core, it involves assigning roles to different AI agents for software creation, granting them a degree of autonomy. The key is that this system must be measurable, closed-loop, and tunable over time. Zach Lloyd emphasizes that a factory should be defined 'through code,' using formats like YAML to specify everything from repositories and skills (MCPs) to integrations and workflows. This code-driven approach allows for explicit definition, making the factory an orchestratable and measurable entity, rather than an abstract concept. This contrasts with common misconceptions where companies might simply string together various automations and call it a software factory.
Interacting with the factory through existing tools
Users, primarily product engineers and engineering managers, can interact with the software factory through familiar tools, integrating it into their existing workflows. This can include sending messages via Slack, creating tasks in Linear or Jira, communicating through GitHub, or making API calls. The analogy used is submitting a work item into the factory, which then initiates a predefined, code-defined workflow. This approach aims to bring the factory's capabilities into the places where developers already work, reducing friction and encouraging adoption. The goal is to make the factory accessible without forcing users into entirely new interfaces for every interaction.
The power of centralized observability and measurement
A significant benefit of this factory approach is the creation of a centralized system for observing and measuring the work of coding agents. This provides a real-time view of how teams are utilizing their factories, capturing conversations, artifacts, and work records. This 'closed-loop system' is crucial for improvement. For instance, agent traces can record entire conversations, including computer usage screenshots, providing a detailed history of the development process. This level of centralization is often missing when engineers primarily work with individual local setups. By aggregating this data, teams can gain insights into their overall output, measure metrics like human intervention per PR, throughput, and costs, and understand where resources are being spent in the Software Development Life Cycle (SDLC).
Implementing AI agent evaluation and quality scoring
Once agent activities are logged, they can be rigorously evaluated. This is done through 'scorers' or evaluators, which can be LLMs acting as judges. These evaluations can assess various aspects, such as how talkative agents are, their effectiveness, and the quality of the code they produce. The system allows for customizability, enabling companies to define specific failure modes they want to track, such as an agent introducing changes that break the build. This evaluation process, which can be configured via code, allows for the creation of smart agents that identify areas for improvement, like agents that might write unnecessary tests, which consume tokens and bloat the codebase. The core idea is to move beyond guesswork and apply engineering rigor to AI agent performance.
The 'Happy Stack': Essential components for a software factory
Building a software factory involves several key infrastructural layers. At the base is the explicit definition of the factory 'as code,' providing versioning, branching, and rollback capabilities similar to traditional SDLC tools. This foundational layer enables self-improvement loops where agents can suggest changes to the factory definition itself. The next layer is data aggregation—collecting agent conversations, artifacts, and interaction records to enable testing, debugging, and performance optimization. This data is crucial for understanding the full lifecycle of a task, from initiation in Slack to a merged PR and potential post-deployment issues. Additionally, an MCP management layer is needed to control access to different skills and tools based on team roles (e.g., marketing, sales, engineering). A compute layer, which can be pluggable to support various sandboxes and VMs (like Mac VMs for iOS testing), is also essential. Finally, an improvement layer, which uses the collected data to refine agents and workflows, and an orchestrator that interprets the factory definition code to execute tasks, are critical components. Access points, like Slack or task trackers, connect the factory to the existing developer ecosystem.
Cost reduction and performance gains through optimization
Warp's internal experience highlights the tangible benefits of optimizing a software factory. Initially, using powerful models like Opus for all tasks led to costs of around $80 per PR. By tuning models, optimizing sub-agent usage, and implementing more strategic agent configurations, they reduced this cost to approximately $15-20 per PR. This optimization also led to a significant increase in throughput, with a 40% rise in PR throughput in one month without an increase in the revert rate. This demonstrates the power of applying engineering principles—observation, measurement, and tuning—to AI-driven development processes.
Benchmarking for informed model and harness selection
A crucial aspect of factory engineering is benchmarking. This involves setting up a suite of tasks, such as front-end coding, and running them against various coding agents and configurations. The goal is to generate company-specific results that balance cost against accuracy. For example, benchmarks might show that a more advanced model like GPT-5-6 achieves 95% accuracy at $6 per task, while a less expensive model like GLM-3.5 achieves 93% accuracy at $2 per task. These insights are invaluable for developing intelligent model routing strategies, ensuring that the most cost-effective and accurate tools are used for specific workflows and codebases within the company.
Who should build a software factory?
The decision to build or adopt a software factory infrastructure depends on team size and complexity. While the concept is appealing, it's generally not cost-effective for very small teams (under 5-10 engineers), who might be better served by direct API usage or off-the-shelf coding agent tools. However, for rapidly growing startups and larger organizations with 10-100+ engineers, a software factory becomes increasingly beneficial. It allows for scalability, customizability, and the ability to adapt to evolving AI models and agent capabilities. The 'happy stack' integrations with tools like Slack and Linear are particularly effective for mid-sized teams. For very large companies with specialized infrastructure, building their own solutions might be an option, but for many, a platform like Warp provides a more accessible path to achieving these advanced engineering capabilities.
Mentioned in This Episode
●Software & Apps
●Companies
●Concepts
●People Referenced
Model Cost vs. Accuracy Comparison
Data extracted from this episode
| Model | Accuracy (%) | Average Cost ($) |
|---|---|---|
| GPT-5 6 | 95 | 6 |
| GLM-5 3.5 | 93 | 2 |
Common Questions
A software factory is essentially a generative system in the cloud where different agents are assigned roles for building software for a team or product. It should be scalable, closed-loop, and allow for tuning, measurement, and benchmarking over time to ensure the most efficient development process.
Topics
Mentioned in this video
A task tracker that can be used to interact with the software factory.
A potential compute provider that could be integrated into the software factory stack.
A tool for software development that has successfully integrated various components into a cohesive product.
Mentioned as an example of a standalone coding agent.
A communication tool used as an entry point for interacting with the software factory, with about 60% of work starting here.
Used as an analogy for defining infrastructure as code, similar to how software factories should be defined.
A task tracker that can be used to interact with the software factory.
A potential compute provider that could be integrated into the software factory stack.
A potential compute provider that could be integrated into the software factory stack.
A model previously used by Warp.dev that contributed to high PR costs, illustrating the need for optimization.
Warp's primary compute interface, launched at AI Europe.
A coding agent that can be benchmarked against other models and harnesses.
Required for testing iOS apps, highlighting a specific compute need for certain development tasks.
A tool mentioned alongside Cognition, implying a similar approach to bundling development components.
A platform for code hosting and version control, used to interact with the software factory.
The company developing the software factory infrastructure stack discussed in the video.
An AI company whose models may be used for inference, with pre-existing contracts companies might need to fulfill.
A company with a highly specialized infrastructure that may not be a suitable customer for Warp's offerings.
Associated with Brandon, a speaker at AIE.
An AI company whose models may be used for inference, with pre-existing contracts companies might need to fulfill.
An AI company whose models may be used for inference, with pre-existing contracts companies might need to fulfill.
A company mentioned as an advisor, developing tools that bundle various software development components.
A company with a highly specialized infrastructure that may not be a suitable customer for Warp's offerings.
Refers to harnesses that can be orchestrated within the software factory framework.
Referenced as a less structured approach that teams eventually move away from towards more platform-based systems.
Business-to-Business Software as a Service, the context for the CTO's question about the necessity of the full stack.
Continuous Integration/Continuous Deployment, a foundational concept in software development that the software factory stack builds upon.
More from Latent Space
View all 272 summaries
68 minBuilding Legal Agents When You Can't See the Data — Jacob Lauritzen, CTO of Legora
28 minBuilding ChatGPT for Finance — Joseph Kim, OpenAI's Head of Financial Services
27 minUsing Agents for Long-Horizon Financial Work — Aman Singh, Arca
107 minHow Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free