Key Moments
Podcast Crossover: AIE, AGI, frontier lab strategy with @matthew_berman and @swyxtv
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI engineers are crucial as model capabilities outpace widespread deployment, creating a persistent 'overhang' ripe for startup innovation, but experts caution against relying on model routing as a winning strategy.
Key Insights
The AI Engineering (AIE) conference was founded on the observation that AI was poised to become a professionalized field like front-end or data engineering, complete with dedicated conferences and tech stacks.
Emerging AI inference chips from companies like Etched are optimizing for post-transformer architectures, but the risk of model architecture changes remains, though current models like GPT-3.5's have shown surprising stability.
Anthropic's Fable 5, while extremely smart, is noted for its slowness and occasional false refusals, leading to a narrative that it's 'nerfed' despite the company's assurances.
OpenAI's rumored 5% equity stake offer to the US government is seen by some as a move to align with the current administration and a potential precursor to government involvement akin to Singapore's sovereign wealth funds.
The probability of long-term AI alignment is considered low by some, with a 'doomer' perspective placing the risk of existential threat within the next 50,000 years, though shorter-term P(doom) estimates vary.
Agent labs, which focus on applying AI to specific problems for clients rather than general model development, are seen as a sustainable strategy for founders, leveraging capability overhangs and providing specialized, trusted AI solutions.
The genesis and evolution of AI engineering conferences
The AI Engineering (AIE) conference emerged from a perceived industry shift, akin to the professionalization of front-end and data engineering. The founder, having spoken at numerous conferences, recognized the burgeoning need for a dedicated space for AI practitioners. Initially, the AIE conference faced challenges in attracting attendees, as the field was primitive, largely focused on prompt frameworks and basic RAG techniques, with agents not yet a prominent feature. A key early success was securing participation from major labs like OpenAI, establishing AIE as a neutral ground for competition and a valuable venue for engineers to access the latest developments across the AI stack.
The rise of specialized inference hardware
The discussion touched upon new players in the AI hardware space, such as Etched, which are developing specialized chips for AI inference. These chips are designed to optimize for workloads that have emerged post-Transformer architecture. While companies like Cerebras and Groq established the market for AI-specific hardware over a decade ago, newer entrants are focusing on the demands of the latest models. The risk of custom chip designs becoming obsolete due to rapid model architecture changes was raised, but the current iteration of models, exemplified by the relative stability of GPT-3.5's architecture, suggests that existing, well-performing models remain in use, offering a degree of stability for hardware investments.
Anthropic's Fable 5 and the debate around model performance
Anthropic's Fable 5 model was a significant point of discussion, with initial excitement tempered by observations of its slowness and occasional 'false refusals' or downgrades in performance. While some users reported a perception of the model being 'nerfed' after its initial release, the exact extent of any degradation and the reasons behind it remain unclear, with Anthropic attributing performance variations to safety mechanisms and ongoing development rather than compute limitations. The inference speed and token usage also present practical challenges, making it suitable for complex problems but less so for everyday tasks. This highlights a common tension between model capability and usability, suggesting a need for continued innovation in inference efficiency.
OpenAI's strategic engagement with the US government
The prospect of OpenAI offering a 5% equity stake to the US government was analyzed as a potentially strategic move. This aligns with OpenAI's historically more cooperative relationship with the White House compared to other frontier labs. The idea is framed not just as a political maneuver but as a recognition that in countries where advanced AI is developed, citizens (represented by the government) should have a stake in the economic upside. Drawing parallels with Singapore's model, where state-owned investment firms hold significant stakes in critical industries, the discussion explored whether this move signals a path towards treating AI more like a utility, though the volatility of the technology suggests it's too early for stringent utility-style regulation. It's speculated that this could be an 'art of the deal' tactic, initiating with an extreme offer that could be rolled back, potentially to expedite model releases or influence regulatory approaches.
Existential risk and AI alignment timelines
A significant portion of the conversation delved into the existential risks posed by advanced AI. One perspective presented a 'doomer' outlook, estimating the probability of AI-related extinction to be as high as 90% on a 50,000-year scale, emphasizing that humanity does not inherently have a right to continued existence and that birthing a new, faster-evolving life form could lead to unfavorable outcomes. Shorter-term 'P(doom)' estimates for 10 and 50 years were also discussed, generally falling much lower. The consensus leaned towards the default assumption being that AI alignment is difficult, and any missed alignment would likely be in the 'safe' direction. The importance of engineers actively monitoring and controlling AI development was stressed as a more pragmatic approach than purely theoretical discussions found in communities like the Effective Altruism (EA) movement.
The limitations of current LLMs and the need for new paradigms
While Large Language Models (LLMs) enable recursive self-improvement (RSI) to some extent, they are seen as limited in their ability to discover truly novel concepts beyond their training data. The current learning paradigms, requiring trillions of tokens for human-equivalent labor, are considered highly inefficient compared to human learning on millions. This inefficiency highlights a key problem that needs solving, potentially through advances in data efficiency and continual learning, to enable agents that can build real-world models and adapt beyond the pre-train/post-train paradigm. Models like Fable are seen by some as potentially marking the end of the current LLM era due to their inherent slowness and cost, signaling a need for different architectures or approaches.
Agent labs as a sustainable business model
The concept of 'agent labs' — companies that focus on solving specific customer problems using AI, rather than building general models — was proposed as a durable strategy for founders. Instead of chasing the latest model capabilities, agent labs aim to be the trusted AI partners for specific industries or professions (e.g., lawyers, dentists). This approach leverages the 'capability overhang' – the gap between peak model capability and its real-world, widespread deployment. While models generalize, agent labs can adapt to new capabilities, applying them to their niche. This contrasts with frontier labs, which often lack the dedicated engineering and customer support resources that agent labs can offer, providing crucial last-mile solutions and integrations. Companies like Cognition, which scored well on 'food bench' (an internal benchmark mentioned humorously), exemplify this specialized approach.
The debate on multi-cloud vs. single-cloud AI strategies
The discussion on model routing and multi-agnostic companies revealed a divide. While many see value in model-agnostic approaches, especially given token budget limitations and the desire to leverage diverse model strengths, the argument was made that historically, significant wins have come from going 'all in' on a single platform or technology. Companies that pride themselves on routing may never fully exploit the unique capabilities of any single model. Furthermore, a single-stack approach can lead to economies of scale and deeper optimization, akin to the cloud computing wars. The marketing of 'routing' was questioned, with a preference for agent labs focused on maximizing the prompt surface area, tool use, and caching of specific, powerful models.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Books
●Concepts
●People Referenced
Common Questions
The AIE conference originated from the observation of industry shifts, similar to front-end and cloud engineering. The founders bought the domain AI.engineer, wrote a blog post, and gained endorsement from Andrej Karpathy, marking a significant moment for the AI engineering field.
Topics
Mentioned in this video
Mentioned as having placed AI at the peak of its hype cycle two years prior to the discussion.
Discussed in relation to OpenAI potentially offering it a 5% equity stake, and its role in regulating AI.
A sovereign wealth fund of Singapore, mentioned as an example of government ownership in key national companies.
A sovereign wealth fund of Singapore, mentioned as an example of government ownership in key national companies.
Mentioned as a major AI lab that participated in the AIE conference, and as potentially offering a 5% equity stake to the US government.
A company that recently emerged from stealth, developing custom AI inference chips.
A major player in GPUs, discussed in the context of custom AI inference chips potentially competing or coexisting with their offerings.
A company that has been developing AI chips for over 10 years, mentioned as a predecessor to newer custom inference chip designs.
Mentioned alongside Etched as a company developing new generation AI chips optimized for post-Transformer architectures.
The company behind Fable, discussed regarding user complaints about false refusals, performance, and the narrative around compute limitations.
Mentioned briefly in relation to the rollout of new models and safety considerations.
A company mentioned as a potential provider of future AI technology beyond the current LLM paradigm.
Mentioned as a potential provider of future AI technology beyond the current LLM paradigm.
Referred to as an example of an 'agent lab' for their field.
A company discussed as an example of an 'agent lab', with insights from someone who spent six months there.
Referred to as an example of an 'agent lab' for their field.
A financial institution mentioned as a client of Cognition, highlighting the support and integration services provided by agent labs.
A code hosting platform, mentioned in comparison to other version control systems used by clients of agent labs.
Mentioned as a type of AI chip, similar to Cerebras, that newer custom chips like Etched are evolving from.
Mentioned as an example of an AI model architecture that has remained relatively stable, making custom chip bets on this architecture reliable.
An AI model that has recently been re-released, with discussions about its performance, potential 'nerfing', and slowness.
Amazon Web Services, mentioned as an example of a cloud provider where concentrating on one platform can yield economies of scale.
Mentioned in the context of the timeline of AI development and regulation, comparing it to early days of electricity.
Large Language Models, discussed as a tool for recursive self-improvement but limited in discovering true 'unknown unknowns' or novel innovation.
Mentioned as potentially being the end of the current era of LLMs due to its slowness and high token usage.
An older generation of model where the capability overhang is still considered real and critical.
A website where one can find a piece discussing the 'Agent Lab' strategy.
Mentioned as an 'agent lab' or tool that could be used by AI engineers.
Referred to as an example of an 'agent lab' for their field.
Mentioned in the context of integration services provided by agent labs like Cognition.
Google Cloud Platform, mentioned as an example of a cloud provider where concentrating on one platform can yield economies of scale.
Mentioned as an open-source project that a fork of Bitbucket was based on, relevant to client integration needs.
Mentioned as a model that might not provide the specific integration services that agent labs like Cognition do.
A foundational AI architecture that newer custom chips are designed to optimize for, unlike older designs.
The human species, discussed in the context of long-term survival timelines and the potential for humanity to be superseded by new life forms like AI.
A strategy for founders to focus on being the AI expert for a specific industry or problem, rather than just building on hot trends.
More from Latent Space
View all 235 summaries
60 minThe 100,000 Sandbox Problem — Akshat Bubna, Modal CTO
109 min🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"
24 minThe Blueprint for Autonomous Work Agents | Gavriel Cohen, NanoClaw
42 minCooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free