Key Moments
Poolside’s Model Factory, Laguna S, Open Models, and the Race to AGI — Eiso Kant, Poolside AI
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Poolside AI's L2 model trains in 8 weeks, outperforming larger models through enhanced behaviors and persistence, not just scale. They champion an open-source future with 100+ foundation model companies, prioritizing democratized access over an intelligence monopoly.
Key Insights
Poolside AI's Laguna S2 model was trained and released in just 5 weeks, while a subsequent model was completed in 8 weeks, highlighting their highly efficient 'Model Factory' process.
The Laguna S model, with 118 billion total parameters and 8 billion active parameters, achieves state-of-the-art performance in its weight class and even surpasses models twice its size, particularly in coding tasks.
Eiso Kant, Poolside AI's co-founder, strongly believes that Large Language Models should operate more like code-writing entities within a virtual machine rather than relying on numerous predefined tool calls, as seen in Laguna S's capabilities.
Poolside AI prioritizes open research and open weights to foster a world with 100 foundation model companies, rather than a few dominant players, believing this leads to greater abundance and choice.
The 'Model Factory' approach at Poolside AI treats model building as an industrialized, end-to-end engineering process, optimizing for the speed of an idea from researcher to experimental result.
Despite the focus on core language models, Poolside AI recognizes the importance of multimodality, especially visual understanding, but maintains a focused path on reasoning and long-horizon tasks as their primary drivers towards AGI.
The genesis of a model factory and a commitment to open research
Eiso Kant, co-founder of Poolside AI, traces his journey into AI back to Andrej Karpathy's 2015 article on recurrent neural networks, which inspired him to focus on language models for code. His early startup, Sourced, invested years and $12 million in open-source models on code, but failed to gain traction in a market that wasn't ready. The subsequent rise of models like ChatGPT felt like vindication. At Poolside AI, founded with a vision of AGI and a world of abundance, the initial premise was that AI capabilities would continue to compound, and reinforcement learning would be key. Initially unfazed by the open-source conversation, they focused on building their technology from scratch. However, as 2023 began, a realization struck: the industry was heading towards a concentration of power, akin to a dystopian sci-fi narrative. This prompted a soul-searching decision to embrace open weights and open research, driven by the belief that a world with 100 foundation model companies is preferable to one with just a handful, even if Poolside were one of the few.
The Model Factory: Industrializing AI development
Poolside AI views model building as fundamentally 90% engineering, a stark contrast to the often-patchy 'spaghetti code' infrastructure prevalent three years ago. Their solution is the 'Model Factory,' an industrialized, end-to-end process transforming raw data into released models. This factory comprises thousands of meticulously engineered components, akin to understanding the entire development history of a manufacturing plant like Foxconn. The core metric being optimized is the speed of an idea from a researcher to a trustworthy experimental result, enabling rapid iteration. With less than 70 researchers and 35 engineers, Poolside is running 10,000-20,000 experiments monthly. This industrial approach has drastically reduced model development time, with Laguna S2 taking only 5 weeks from pre-training to launch, and another model following in 8 weeks. This efficiency allows for continuous improvement and the immediate repurposing of compute for new models, treating model releases not as singular monumental tasks but as regular outputs of a robust factory.
Laguna S: A leap in behavior and efficiency
The Laguna S model, an 118 billion parameter model with 8 billion active parameters, exemplifies Poolside's progress. It was developed in a highly efficient manner, fitting on a DGX Spark and running at 30-40 tokens per second. What sets Laguna S apart, according to co-head of applied research Peng, is not just raw intelligence but 'different behavior'—more verification, less taking things for granted, and increased persistence. This emphasis on behaviors like reasoning and problem-solving, rather than solely scaling parameters, is seen as crucial for future advancements, especially in tasks requiring long horizons and complex problem-solving. The model has demonstrated remarkable capabilities, such as independently solving Erdos 397 and creating a Wi-Fi scanner without external libraries. This persistence and improved reasoning have led Poolside to use Laguna S extensively internally, highlighting its potential for knowledge work and coding tasks.
Rethinking tool use: Embracing code generation
Eiso Kant expresses a strong skepticism towards the current paradigm of using numerous 'tool calls' within models. He argues that for complex, long-horizon tasks, models should operate within a virtual machine, writing code to interact with systems and data directly. This approach, evident in models like Laguna S and frontier models, offers greater freedom and efficiency than chaining multiple predefined tools. Models, when trained to write code, naturally employ conditional logic, loops, and other programming constructs, which is more efficient and generalizable than relying on a limited set of pre-programmed tools. Poolside's philosophy is to provide models with a minimal harness and a containerized environment with necessary tools and data access, allowing them the freedom to solve tasks most efficiently through code generation.
The power of open research and democratized access
Poolside AI is committed to open research and open weights, believing it is the most meaningful contribution to democratizing AI. They see open releases not just as providing model weights but as sharing the lessons learned from tens of thousands of experiments. This approach, they argue, is crucial for fostering a diverse ecosystem of foundation model companies, preventing a concentration of intelligence in a few hands. While acknowledging the challenges, such as potential misuse and governmental response, their core conviction is that a world with 100 foundation model companies offers greater abundance and choice than one dominated by a few. They actively encourage researchers to start competing foundation model companies, aiming to speed up progress and ensure broader access to AI capabilities.
The future of AI: Beyond next-token prediction and toward embodied intelligence
The conversation touches on the evolving landscape of AI training. While pre-training remains critical, Eiso Kant suggests a paradigm shift where reinforcement learning (RL) might be integrated earlier in the training process, moving beyond simple next-token prediction. The massive dataset of the web, containing humanity's knowledge, is still underutilized; the focus is on teaching models to think earlier. Distillation and numerous RL environments are current 'drugs,' but the goal is to extract more value from the web itself. Mid-training is viewed as a form of staged pre-training, a necessary compromise due to computational limitations, but the ideal is a continuous, optimal curriculum from token zero. Poolside is investing heavily in research to teach models to 'think' earlier, a path they believe holds immense potential for advancing AI capabilities toward AGI.
Navigating the hardware and regulatory landscape
The discussion highlights the symbiotic relationship between hardware and AI development. NVIDIA's advancements in GPUs and networking are seen as critical enablers of progress, allowing for larger models and faster iteration. Poolside AI, while acknowledging the difficulty of building hardware companies like NVIDIA, emphasizes the need for a level playing field where aspiring foundation model companies have access to the necessary tools. They caution against regulatory actions that could stifle innovation or create monopolies, drawing parallels to how restricting cigarette advertising inadvertently benefited existing tobacco companies. The ideal future, they believe, is one where intelligence becomes a commodity, accessible and affordable, fostering competition and choice rather than a concentrated power structure. This includes enabling open models, as restrictions could hinder innovation and lead to a dystopian future.
The critical role of agency and constraints in innovation
When discussing team productivity and hiring, 'agency' emerges as a paramount quality. High-agency individuals, those who proactively identify and pursue opportunities, are highly valued. Poolside AI's success is attributed to its team of high-agency individuals who are aligned with a common mission and operate within defined boundaries. These constraints, rather than hindering progress, often force innovation, as seen in Poolside's relatively lower compute and data costs compared to competitors. The company is actively hiring for applied research and engineering roles, emphasizing the unique opportunity for individuals to have a significant impact within a smaller, mission-driven organization. The call to action is for those who are aligned with their mission and optimize for impact to join their journey, contributing to a future where intelligence is democratized and accessible.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Books
●Concepts
●People Referenced
Common Questions
Eiso Kant was inspired by Andrej Karpathy's 2015 article, 'The Unreasonable Effectiveness of Recurrent Neural Networks,' which led him to pivot his startup overnight to focus on RNNs, LSTMs, and transformer models for code generation. This early work fueled his belief in open-sourcing AI capabilities.
Topics
Mentioned in this video
An article by Andre Karpathy that inspired the speaker to get into language models and coding for AI.
A paper mentioned for demonstrating early reasoning induction in models during pre-training, seen as a public example of work Poolside AI had pursued privately.
A neural network architecture that outpaced earlier models, mentioned in the context of the speaker's early work on language models.
A low-precision training method that the speaker is excited about adopting for future model training.
The core API for Wireless Local Area Networks, which Laguna S was able to figure out to create a Wi-Fi scanner without internet access.
One of the companies highly respected by the speaker for their confidence in scaling up language models.
One of the companies highly respected by the speaker for their confidence in scaling up language models.
Noted by the speaker as not initially holding the opinion on reinforcement learning's importance, but now having significant financial success.
The speaker's current company, focused on enabling a world of abundant AGI and open-source models, emphasizing an engineering-first approach.
A Chinese AI lab and model mentioned for its open-source innovation and as a competitor to Western models.
The original name intended for Poolside AI, inspired by the 'snowball effect', but it was a trademark of Amazon.
Mentioned as the company that trademarked 'Snowball Apps', preventing Poolside AI from using the name.
Mentioned in the context of a potential deal similar to the OpenAI-Microsoft partnership, which Poolside AI considered before its official incorporation.
An agent harness mentioned as a subject of collaboration and development interest for Poolside.
Credited as a foundational company for the AI industry, enabling progress with its GPUs and networking, although its power in compute allocation is questioned.
Mentioned as a company even further down the supply chain than NVIDIA, producing the chips essential for AI compute.
Its release felt like a vindication of the speaker's earlier work on language models and code.
A new model whose launch day coincided with the podcast, with Poolside's model outperforming it on some benchmarks.
A model mentioned as a benchmark for smaller models, comparing Laguna S to its performance level.
Where Nikolai, the co-head of applied research at Poolside AI, previously worked on language models.
A humorous suggestion for a company name by the co-founder, highlighting the difficulty in naming tech companies.
A computing system that the Laguna S model (118B parameters) can fit and run on efficiently.
A framework mentioned as a comparison point for base models during the early days of open-source AI.
One of the early open-source models available, mentioned in the context of model development history.
An early large language model mentioned for its ability to run on a single H100 GPU.
A benchmark mentioned on which Poolside's model performs well, sometimes achieving state-of-the-art results.
OpenAI's model, whose initial release was controversial due to fears of misuse, used as an example of past overestimations of danger.
An NVIDIA chip mentioned as an example of specialized hardware that works well with GPUs for inference, relevant to optimizing RL training.
The current hardware setup at Poolside AI (10k H200 cluster) for training models, with plans to scale up.
A GPU mentioned for its capability to run large models and as a key component in AI development.
Used as an example of a consumer hardware that can run significant models, highlighting the increasing accessibility of AI.
A platform compared to Poolside in terms of minimal surface area for tooling, reflecting a similar philosophy.
NVIDIA GPU architecture, where the speaker is excited to apply lower precision RL techniques once available.
More from Latent Space
View all 239 summaries
90 minCausal Models Need Causal Data - Xaira’s X-Cell model (Bo Wang & Ci Chu)
50 minThe AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO
49 minWhy AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab
29 minPodcast Crossover: AIE, AGI, frontier lab strategy with @matthew_berman and @swyxtv
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free