Key Moments
The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Modal's CTO explains how their cloud platform, initially for data pipelines but now AI-focused, offers unique solutions for bursty compute, agent workloads, and specialized AI infrastructure, attracting startups disrupting traditional cloud models.
Key Insights
Modal added GPUs a year before ChatGPT and didn't initially consider it a major feature, highlighting the rapid evolution of AI infrastructure needs.
The company shifted its SDK team's focus from 'developer experience' (DX) to 'agent experience' (AX), recognizing that agents also benefit from simplified setup and self-provisioning runtimes.
Sandbox workloads, particularly for Reinforcement Learning (RL), can be incredibly bursty, sometimes requiring up to 100,000 sandboxes simultaneously.
Modal's core differentiator is its autoscaling capability, including GPU snapshotting for faster cold starts, which is crucial for elastic inference and on-demand training.
Modal operates across 17 cloud providers, positioning itself as a 'super cloud' by building its own reliability layer to abstract away provider-specific issues.
The company is open-sourcing its work, such as 'D-Lash' for speculative decoding, to make frontier-level model performance accessible to everyone and contribute to the wider AI ecosystem.
From Data Pipelines to AI Specialization: The Origin Story
Modal began with the vision of creating a better runtime for workflow orchestration, addressing the complexities and limitations of Kubernetes for bursty, compute-heavy workloads. The initial focus was on serverless functions for ETL and job queues, but the company quickly realized the broader applicability of a flexible runtime. A pivotal moment was adding GPU support a year before ChatGPT's release, an addition they initially underestimated. This foresight allowed Modal to build infrastructure primitives that could support the burgeoning demand for AI applications. The core problem they aimed to solve was the poor developer experience (DX) associated with traditional cloud platforms, especially for custom environments, accelerators, and rapid scaling up and down.
Developer Experience and the Shift to Agent Experience
A key early innovation for Modal was the use of decorators to define infrastructure requirements directly within the code, bypassing the need for extensive YAML configuration. This approach significantly simplified the developer experience, allowing for more dynamic and expressive code operations. However, as AI agents have become more prevalent, Modal has shifted its SDK team's focus from 'developer experience' (DX) to 'agent experience' (AX). The philosophy is that agents too benefit from simplified setup and self-provisioning runtimes. Instead of an agent parsing complex Kubernetes files, it can achieve its goals through a few decorator changes, seeing its actions live in real-time. This highlights a fundamental advantage for agents using Modal: collocated infrastructure requirements with the code that runs them, proving faster than operating on other substrates.
Elastic Inference and Specialized Compute for AI Workloads
Modal's most significant use case today is elastic inference, particularly for custom models beyond the typical large language model (LLM) space, serving clients in audio (Sunumo), video (Runway), robotics, and computational biology. The platform excels at scaling inference up or down to meet unpredictable traffic patterns, which often involve multiple models deployed across different regions. To address the slow cold start times typically associated with GPUs, Modal incorporated GPU snapshotting, allowing it to save and restore GPU states, significantly speeding up subsequent cold starts. Beyond inference, Modal also supports on-demand training, batch jobs, and crucially, sandboxes. Reinforcement Learning (RL) workloads, in particular, are highlighted as exceptionally bursty, sometimes demanding up to 100,000 sandboxes concurrently, a scale that traditional cloud infrastructure struggles to meet efficiently.
The Rise of Sandboxes and Agent-Native Primitives
Modal built sandboxes in May 2023, predating the widespread recognition of their importance for AI agents. They quickly found that agents, even those not yet adept at complex tool-calling or self-correction, would diverge or fail after multiple iterations. To counter this, Modal's sandboxes provide a self-provisioning runtime environment where agents can iterate and see changes live. This is particularly useful for agent development where infrastructure needs to be dynamically configured. Even now, when code might be treated as a black box by agents, observability remains critical for manual intervention and decision-making. Modal's approach makes it faster for agents to utilize these environments compared to managing traditional infrastructure.
Open-Sourcing Innovations: Speculative Decoding and Auto Endpoints
To democratize advanced AI capabilities, Modal is actively open-sourcing its innovations. 'D-Lash', a block-based speculative decoding system, is one such example. Speculative decoding uses a smaller 'draft' model to predict multiple tokens ahead, which are then verified by a larger model. This approach significantly speeds up inference by improving compute utilization and achieving multiplicative speedups (2-4x) compared to single-token prediction, without compromising quality. Following this, Modal launched 'Auto Endpoints' to make frontier-level inference performance accessible. These endpoints offer modal's optimizations, including speculative decoding, with a simplified UI or CLI experience, while still allowing users to export the underlying code and customize further, catering to both ease-of-use and advanced customization needs.
A 'Super Cloud' Strategy: Multi-Cloud and Reliability
Modal operates across 17 cloud providers, positioning itself as a 'super cloud' by abstracting away the complexities of heterogeneous infrastructure. This capital-light approach allows them to move rapidly and focus on their software differentiators. A significant investment has been made in building a robust reliability layer on top of various cloud providers, ensuring that user workloads are unaffected by individual provider issues, such as a GPU failing. This capability allows Modal to leverage a wider range of capacity than a single user might access and maintain consistent performance across diverse underlying infrastructure, addressing needs for data locality, low latency, and specific compute configurations.
Networking Advancements: From Overlay Networks to RDMA
Modal's infrastructure development extends to networking, addressing the growing need for inter-container communication and specialized interconnects. They now support multi-container sandboxes (pods) with 'sidecar' functionality, enabling more complex setups like running man-in-the-middle proxies for logging or controlling outbound network traffic. For inter-node communication, Modal has developed 'i6PN', an overlay network using IPv6 addresses that provides private networking for containers within a workspace. Initially built for distributed training, this primitive allows containers to address each other via private IPs without needing explicit network configuration. While i6PN is a TCP overlay, it's also crucial for key exchange for RDMA setups. Modal also supports RDMA networking for distributed training clusters, offering speeds up to 3 TB/s internally, crucial for speeding up data transfer between GPUs in scenarios like post-training medium-sized quantized models.
The Future of AI Infrastructure: Agents, Multi-modality, and Primitives
Modal sees its future in building primitives that simplify AI development across the entire model lifecycle, from data prep and training to inference and deployment. While LLM inference remains a major focus with advancements like auto endpoints and custom model post-training, Modal is also expanding into other verticals like real-time audio/video processing and computational biology, where batch processing or low-latency streaming is key. The company emphasizes that its primitives cater to diverse needs, not just the LLM market. On the agent side, Modal works closely with customers to adapt to rapidly evolving requirements, considering new primitives beyond sandboxes and persistent storage. The firm is skeptical of LM-mediated permissions for sensitive sandbox operations, advocating for hard guardrails, possibly paired with softer, mediated ones. Ultimately, Modal's strategy is to continue building fundamental infrastructure components that enable innovation across the AI landscape, whether for traditional ML, agents, or emerging multi-modal applications.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Common Questions
Modal is a cloud platform built from scratch for AI applications. It addresses the difficulties of managing infrastructure by providing primitives for workloads like inference, training, batch processing, and sandboxing, aiming for a better developer and agent experience.
Topics
Mentioned in this video
Mentioned as a difficult-to-manage platform for workflow orchestration, not built for burstiness or custom images, and having a terrible developer experience.
Modal added GPUs to their product a year before ChatGPT came out, not foreseeing its significant impact.
A cloud platform built from scratch for AI applications, covering inference, training, batch processing, and sandbox workloads, and evolving to focus on agent experience and specialized compute.
Cursor Composer is mentioned as an example of a company performing RL on models every couple hours, requiring scalable compute like Modal provides.
A specific model mentioned as an example for comparison with Modal's inference engine.
Mentioned as an off-the-shelf inference engine for comparison with Modal's capabilities.
Mentioned as an inference engine for comparison with Modal's capabilities.
A successful background agent built on Modal, showcasing the platform's capabilities for reactive and scalable applications.
Mentioned as part of the ecosystem of managed agents, where companies like OpenAI are entering Modal's space.
GitPod rebranded to Owner, and its team's potential integration with OpenAI is noted.
Mentioned as an example of companies building on top of Python, a dominant language in data and ML.
The speaker wrote a post about infrastructure software-defined infrastructure while at Temporo.
Modal does not aim to compete with companies like Renders, focusing instead on workloads that require specialized compute and elastic scaling.
A company where an embedded Modal employee was present, highlighting the close collaboration and integration between Modal and its customers.
Mentioned as a source for models that can be hosted in Modal's GPU sandbox environments.
A company that uses Modal for audio inference, showcasing Modal's capability in serving custom models beyond the LLM space.
Mentioned alongside Fireworks as an inference provider.
A company that announced efforts in continual learning, a workload that Modal might support as its offerings evolve.
A company that uses Modal for its accounting agent, highlighting the need for more control over compute primitives compared to simpler API-based solutions.
Mentioned in the context of GPU types (H200, B200) used in automated research and optimization.
Mentioned again in relation to GitPod's rebranding and potential integration, suggesting a strong future for code execution environments.
Referenced for its technical strength and its market in CI/CD, with potential for agents to drive more CI usage.
Mentioned as recently being acquired by Qualcomm, and its connection to Python tooling.
Acquired Modular, indicating the ongoing consolidation and strategic moves in the tech ecosystem.
Mentioned as a fallen competitor, with Modal's strategy differing by focusing on providing a code-level platform rather than just model APIs.
More from Latent Space
View all 234 summaries
109 min🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"
24 minThe Blueprint for Autonomous Work Agents | Gavriel Cohen, NanoClaw
42 minCooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen
71 minThe Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free