Key Moments
Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal
Latent Space PodcastKey Moments
Modal CEO Erik Bernhardsson explains why AI compute is sold in "rectangles" but should be dynamic, and how Modal's cloud infrastructure provides flexible, on-demand GPU access, leading to 50x growth in two years.
Key Insights
Modal experienced over 50x growth in revenue, users, and usage in two years.
Erik Bernhardsson's prior work at Spotify involved building the first version of their music recommendation system, requiring extensive matrix operations and heavy computation.
Modal's revenue surged from ~$100/month in November 2022 to $30,000/month by January 2023, largely driven by Stable Diffusion workloads.
The company now works with nearly 25 cloud providers and has a dedicated procurement division of four full-time employees to manage capacity.
Modal Sandboxes, launched in 2023, now account for roughly one-third of the company's revenue, enabling complex agent-based applications.
The company is investing in frontend engineers (FDEs) to assist customers with performance optimization and proactively identify cost-saving opportunities, viewing this role as a bridge to future product features.
Founding Modal: Addressing infrastructure gaps learned at Spotify
Erik Bernhardsson, CEO of Modal, outlines the company's mission: to build AI infrastructure providing a compute and storage platform for AI application development. Modal serves clients like Deck of Gone, Suno, Runway, and Cognition, supporting inference, sandboxing, training, and batch processing. Their SDK simplifies deployment and scaling for production environments. Bernhardsson's journey to Modal was heavily influenced by his seven years at Spotify, where he led the development of the initial music recommendation system. This role involved significant matrix analysis and computationally intensive tasks, highlighting a persistent need for better AI infrastructure, which ultimately inspired Modal's creation. He contrasts this with his time at better.com, describing it as a "rollercoaster" and a "side quest" that provided valuable lessons in management, team building, and developer productivity, though less directly relevant to Modal's core infrastructure focus.
From Luigi to Modal: Evolving from workflow orchestration to cloud execution
Bernhardsson's earlier work on Luigi, an open-source workflow orchestration framework at Spotify, provided foundational insights. While Luigi focused on scheduling, Bernhardsson realized the problem extended to the execution of code in the cloud. This led to the development of Modal, initially envisioned as a general-purpose platform for running code – be it GPU-intensive AI/ML tasks or data processing – in the cloud. A key realization early on was the necessity to build a custom system, including a file system and container runtime, to effectively manage the complexities of AI workloads. This approach contrasts with generic cloud solutions that often fail to meet the specific demands of modern AI, particularly the efficient management of expensive, high-utilization GPUs and the loading of massive models.
The Stable Diffusion catalyst and founder-led marketing
The launch of Stable Diffusion in late 2022 proved to be a pivotal moment for Modal. While ChatGPT gained significant attention, Stable Diffusion emerged as the killer application that validated Modal's platform for general AI inference. Bernhardsson attributes a significant portion of Modal's early growth to his active presence on Twitter and blogging, a strategy he terms "founder-led marketing." By consistently sharing his work, insights on cloud computing, and the development of Modal, he cultivated a following and generated excitement. This organic approach, leveraging his personal brand and content creation, was instrumental in attracting initial users and establishing Modal's presence in the AI infrastructure space.
Rapid revenue growth and the concentration of early demand
Modal's revenue trajectory saw a dramatic acceleration starting in late 2022. From approximately $100 per month in November 2022, revenue jumped to $2,000 in December and then to $30,000 in January 2023. This explosive growth was heavily concentrated, with a single New York-based company in the Stable Diffusion space, "Mag," accounting for 80-90% of revenue in Q1 2023. While this concentration presented risks, Modal successfully diversified its revenue streams over time, with "Suno" becoming another significant client in late 2023. The company now boasts a well-diversified revenue base, reflecting its expanding client portfolio and the broad applicability of its platform.
Reimagining cloud infrastructure for AI workloads
Bernhardsson critiques traditional cloud tools like Kubernetes and Docker, designed for stable, low-utilization backend workloads, as ill-suited for the demands of AI. The scarcity and high cost of GPUs, coupled with the need for rapid scaling and efficient model loading, necessitate a reimagined infrastructure. Modal has invested heavily in building its own systems, including a custom file system, auto-scaling scheduler, and capacity management tools. These are designed from the ground up to handle the unique challenges of AI, such as managing fluctuating GPU demand, minimizing idle time, and optimizing the transfer of massive models between CPUs and GPUs. This deep integration allows Modal to offer more efficient and cost-effective solutions compared to retrofitting older technologies.
Navigating the GPU capacity crisis: From spot instances to long-term commitments
The rapid growth in AI has created immense pressure on GPU supply. Modal initially relied heavily on spot and on-demand instances from major cloud providers like AWS, GCP, and Azure, as well as early capacity from Oracle for specific hardware needs. However, the escalating demand and changing cloud market dynamics forced Modal to adapt. The company now engages with approximately 25 cloud providers, often securing capacity through three-year deals with significant upfront payments and fixed pricing. This shift has necessitated the creation of a dedicated procurement department and a full-time engineering team focused on managing physical server capacity globally, addressing network issues, faulty drives, and GPU detachments. The strategic decision of whether to own or lease compute is a complex trade-off involving capital efficiency, operational complexity, and financial risk, with Modal remaining open to various models, including potential future acquisitions.
Dual focus on compute capacity and developer experience
Modal's value proposition rests on two pillars: accessible compute capacity and a superior developer experience. The company's ability to aggregate and finely partition global compute capacity, supported by rapid scaling and snapshotting capabilities, allows for flexible, on-demand resource allocation that contrasts with the rigid, three-year contracts often dictated by the market. This elasticity addresses the mismatch between static "rectangle" capacity sales and the dynamic, fluctuating needs of AI workloads. The developer experience is enhanced through features like serverless functions (Modal Functions), which simplify deployment and scaling, and Modal Sandboxes, providing isolated environments for code execution. Bernhardsson emphasizes that factors improving developer experience, such as clear documentation and fast feedback loops, also benefit agent-based workflows, creating a synergistic effect.
Diversification into agent sandboxes and future product strategy
Modal Sandboxes, initially launched in 2023, have become the company's second major product, now contributing roughly a third of revenue. Their growth has been fueled by the increasing sophistication of LLMs in code generation and the rise of agent-based applications. Modal's existing infrastructure for container management and isolation proved adaptable for these use cases. Looking ahead, Modal is optimistic about several areas: AI training (particularly for small to medium-sized models), batch processing for large-scale data tasks, and low-latency applications like speech-to-text. The company views itself as a provider of foundational AI building blocks, enabling customers to construct novel applications that were previously impossible. While AI is becoming ubiquitous, Modal focuses on providing robust, general-purpose components for developers.
Embracing alternative accelerators and the evolving hardware landscape
While NVIDIA GPUs remain dominant, Modal is exploring alternative accelerators. Bernhardsson predicts that 2027 might be the year non-NVIDIA GPUs gain significant traction, citing advancements from AMD and Google's TPUs. The bottleneck isn't just chip manufacturing (TSMC, ASML) but also the software layer. Companies like Cerebras and Groq are showing promise, with potential for hybrid inference models that leverage different accelerators for specific tasks. Modal prefers to source smaller, diverse capacity pools globally, seeing opportunities where larger players might overlook them. This approach aligns with a long-term view that hardware monopolies are unsustainable, and market diversity will ultimately benefit the ecosystem.
Security as a shared responsibility and the role of AI agents
The "hugging face incident," where a compromised application on Modal gained internet access, highlighted the shared responsibility for security. Bernhardsson clarifies that Modal itself was not breached; rather, a third-party application running on the platform was exploited. This experience led Modal to consider implementing more restrictive default security settings, similar to AWS's approach with S3 buckets. They are also leveraging AI agents internally for cybersecurity, tasking them with attempting to "escape" sandboxes to proactively identify vulnerabilities. This approach, akin to a "room escape" game for agents, aims to bolster security by simulating real-world attack vectors, thereby enhancing both defensive capabilities and the robustness of their platform.
The enduring importance of developer experience and the future of infrastructure
Modal remains committed to an exceptional developer experience, recognizing that factors enhancing it for humans—like clear documentation, fast feedback loops, and robust tooling—also benefit AI agents. The trend towards more programmable infrastructure, where configuration is managed via SDKs rather than declarative files, is crucial. Bernhardsson believes this shift makes infrastructure more accessible and manageable. The company is adapting its product roadmap based on internal testing with agents, ensuring backward compatibility and thoughtful feature deprecation. The focus is on providing general-purpose building blocks that empower developers to create novel applications, reflecting a belief that the future of software lies in abstracting away infrastructure complexities.
Strategic product development and building for the next era of AI
Bernhardsson identifies three key focus areas for Modal: 1) managing the financial complexities of compute capacity procurement and pricing, 2) advancing the product strategy by identifying the next crucial building blocks for AI applications, and 3) nurturing a strong company culture as the team grows rapidly. He emphasizes the need to anticipate future demands, drawing parallels to Modal's early work on serverless functions and sandboxes. The company is actively exploring areas like training, low-latency applications, and agent frameworks. The long-term vision is to provide a comprehensive suite of general-purpose AI infrastructure components, allowing customers to innovate and build applications unimagined today.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Common Questions
Modal provides AI infrastructure, including a compute and storage platform that enables customers to build and deploy AI applications like inference, testing, training, and batch processing.
Topics
Mentioned in this video
A company building AI infrastructure, providing a compute and storage platform for customers to build AI applications.
An AI model that became a prominent application for Modal's platform, driving significant early growth.
A parallel computing platform and API model created by Nvidia, mentioned in the context of system challenges.
A customer of Modal, using their platform to build AI applications and contributing to a significant portion of their revenue.
An operating system for which Modal's sandboxes now offer full compatibility.
A container orchestration system mentioned as an example of foundational tools for a new technology era.
An AI model used by Modal for programming assistance.
A New York-based company that was Modal's largest customer in early 2023, representing 80-90% of their revenue.
A major cloud provider used by Modal for capacity.
A major cloud provider used by Modal for capacity.
An open-source workflow orchestration framework developed by Erik Bernhardsson at Spotify, related to tools like Airflow and Prefect.
A workflow orchestration tool mentioned in the context of Luigi.
A workflow orchestration tool mentioned in the context of Luigi.
A CI/CD platform mentioned in the context of optimizing parallelization in testing.
A workflow orchestration tool mentioned in the context of Luigi.
An AI model mentioned as a significant event around late 2022, though Stable Diffusion was more directly impactful for Modal's initial growth.
A user interface for Stable Diffusion, mentioned in the context of how users were interacting with Modal.
A containerization platform mentioned as an example of foundational tools for a new technology era.
An AI model used by Modal for programming assistance.
A major cloud provider initially used by Modal for capacity, and later used as a benchmark for security practices.
Amazon S3 storage, mentioned in the context of security lessons learned from public bucket configurations.
An open-source tool used by Modal for programming assistance.
A sandboxing tool used by Modal for container isolation.
An open-source model mentioned as being competitive with leading models at a lower cost.
An open-source model mentioned as being competitive with leading models at a lower cost.
A hardware vendor Modal could source equipment from.
A customer of Modal, using their platform to build AI applications.
Erik Bernhardsson's previous employer where he worked on building the first music recommendation system, including features like Discover Weekly.
A major AI company and a competitor for GPU capacity.
A cloud provider mentioned as a beneficiary of GPU scarcity.
A startup attempting to create pricing indices and trade derivatives for computing resources.
A customer of Modal, using their platform to build AI applications.
Erik Bernhardsson's previous employer where he served as CTO, growing the technical team and learning about developer productivity.
A company mentioned as a user of Modal, contributing to the growth of Modal Sandboxes.
A customer of Modal, using their platform to build AI applications.
A startup attempting to create pricing indices and trade derivatives for computing resources.
A cloud services provider mentioned as an example of a system that agents can help configure.
A cloud provider mentioned as an example of modern cloud hyperscalers with impressive fundraising capabilities.
A platform where a security vulnerability was exploited, leading to a security incident involving a Modal application.
A cloud monitoring and analytics company, cited as an example of the success of cloud computing.
A company developing alternative accelerators, mentioned for its performance in certain workloads.
A company using Modal for background agents, contributing to the growth of Modal Sandboxes.
A hardware vendor Modal could source equipment from.
A semiconductor manufacturer mentioned as a bottleneck in the supply chain.
An open-source model mentioned as being competitive with leading models at a lower cost.
A company mentioned in the context of having significant pre-booked capacity, which Modal could potentially leverage.
A company developing alternative accelerators, mentioned as a potential alternative to Nvidia GPUs.
A company involved in semiconductor manufacturing equipment, mentioned as a bottleneck in the supply chain.
A cloud provider Modal rented capacity from early on due to favorable pricing for specific hardware needs.
A key provider of GPUs, discussed extensively in the context of capacity, pricing, and market competition.
A major AI company and a competitor for GPU capacity, also a partner using Modal's sandboxes.
A company whose processors are noted for performing well in certain areas, as an alternative to Nvidia.
A company that uses TPUs internally, mentioned in the context of alternative AI hardware.
A startup attempting to create pricing indices and trade derivatives for computing resources.
A cloud-based data warehousing company, cited as an example of the success of cloud computing.
A concept related to the cognitive limit to the number of people with whom one can maintain stable social relationships, mentioned in the context of company culture and growth.
A field where AI models could be applied for material discovery and property testing.
A field where AI models are expected to see significant growth, due to the ability to perform simulations and gain feedback loops.
A networking concept discussed as part of legacy security models.
A feature Erik Bernhardsson helped build at Spotify as part of the music recommendation system.
A product launched by Modal in 2023, providing isolated environments for running code, which has become a significant revenue driver.
Modal's primary product, launched in 2021, offering serverless functions that are easy to deploy and scale.
More from Latent Space
View all 267 summaries
27 minThe Recommendation Engine for Commodity Traders — Anekha Sokhal, Moshi
76 minHigh Taste Testers: OpenAI Shipped Everything. What Actually Works?
110 minFrom Wet Lab to RL Env: 10,000 Drug-Discovery Campaigns in a Weekend — Ian Quigley, Leash Bio
29 minHow to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free