Key Moments
The AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI models struggle with context, leading to 'context rot' and inaccuracy. Engram is developing 'cartridges' to compress knowledge into model weights, aiming for more efficient and intuitive AI.
Key Insights
Engram's approach compresses knowledge into 'cartridges,' aiming for representations that are a thousand times more compressed than typical textual data.
Current LLMs are akin to first-time chefs reading a textbook for every dish, lacking the intuition of experienced chefs who internalize knowledge.
In 18 months, companies may have trillions of tokens of internal data, making current methods like RAG prohibitively expensive and less accurate due to context rot.
The company is focusing initially on token efficiency and cost reduction for reasoning on large contexts, with long-term goals for continual learning and adaptive AI.
Engram's long-term vision includes personal AI models that improve over time like Tamagotchis, potentially running on personal devices.
Intelligence and efficiency are inseparable; doing more with less is seen as the next paradigm, enabling more ambitious tasks and longer horizons for AI.
The limitations of current AI context and memory
Dan Biderman, co-founder and CEO of Engram, discusses the significant limitations of current AI approaches to handling large amounts of information. He explains that while technologies like Retrieval Augmented Generation (RAG) and long context windows have improved AI's ability to access information, they eventually break down. This breakdown, termed 'context rot,' occurs because as models are fed more data, they become confused and less accurate, even with massive context windows. Biderman likens current LLMs to a chef who must read a cookbook from scratch for every single dish, lacking the intuitive understanding and deeply ingrained knowledge of an experienced chef. This lack of internalized, intuitive knowledge means AI cannot innovate or adapt as effectively as humans, especially when dealing with vast and complex datasets.
Engram's 'cartridge' approach to knowledge compression
Engram's core innovation lies in creating compressed knowledge representations called 'cartridges.' Unlike simply storing raw text or RAG documents, these cartridges are developed through a process similar to model pre-training. The AI is trained to study a large corpus of knowledge, essentially quizzing itself and learning to form compact representations. These cartridges can then be loaded into a model, acting like a 'brain state' that is dramatically more compressed—potentially a thousand times smaller than traditional textual data. This allows AI to operate with far fewer tokens, leading to increased accuracy and reduced confusion. The goal is to imbue models with an 'intuition' that goes beyond rote memorization of recipes or notes, enabling them to infer, generalize, and even innovate.
The evolving data landscape and the need for efficiency
Biderman projects that within 18 months, many companies will possess 'internet-scale' proprietary data, potentially trillions of tokens. This massive influx of data will render current methods of accessing information insufficient and prohibitively expensive. Trying to manage, index, and update a wiki or knowledge base of trillions of tokens would be an immense undertaking. Furthermore, feeding this vast data to frontier models, which start from a point of ignorance about a specific company's data, will consume enormous resources (tokens, compute). The inherent problem of context rot will only be exacerbated at this scale. Engram's focus on token efficiency and creating more compressed knowledge representations is a strategic response to this inevitable data explosion, aiming to prevent models from becoming inaccurate and costly to operate.
Bridging the gap between textual knowledge and intuition
While acknowledging the value of textual representations like notes and diaries—scientists, chefs, and programmers all use them—Biderman emphasizes that this is only one part of intelligence. The crucial missing piece for AI is the 'intuition' derived from a nervous system that processes this information. Human chefs don't just read recipes; they learn through experience, taste, touch, and understanding of ingredients. Engram's training methodology aims to replicate this by creating learned representations that go beyond mere text. The aim is to combine the best of both worlds: the ability to document knowledge (like notes) and the ability to internalize and utilize it intuitively (like a well-trained brain), preventing the need to constantly re-read the same information.
Long horizon agents and the future of AI memory
The challenge of 'long horizon agents'—AI systems that need to remember and act upon information over extended periods—is a key motivator for Engram. Biderman sees continual learning and memory as disguised forms of long context. Current solutions like compaction, which involves models managing their own context by evicting less relevant tokens, are helpful but lossy and can lead to confusion and forgetfulness in deep sessions. Engram believes a neural memory trace, also a form of lossy compression but within weights, is part of the solution. Ultimately, they envision AI systems capable of gradient-based updates during operation (test-time training) to tackle complex, long-duration tasks in science, engineering, and defense more effectively. This paradigm shift moves beyond simply scaling up with more resources to achieving more with less.
The vision for personalized and adaptive AI
Engram's ultimate ambition is for every individual to have their own AI model, or a set of weights, that represents and learns from their unique knowledge and expertise. This personal model would improve over time as its user interacts with it, akin to nurturing a Tamagotchi. While the long-term goal is for these personalized models to run on user devices, Engram is initially focusing on enterprises where AI is already heavily utilized. They are developing parameter-efficient fine-tuning methods and 'cartridge' systems that can represent knowledge effectively and be auditable by users. This approach contrasts with general model providers where user feedback has an unclear impact, offering a tighter loop where user input directly improves their personal model.
Defining the boundaries of AI memory and external knowledge
A central research question for Engram, mirroring the study of human memory, is determining what knowledge should be internalized into AI weights ('internalized') and what should remain as externalizable notes or RAG documents. Biderman notes that excessive memory retention can be distracting and overwhelming, suggesting that a degree of 'forgetting' is healthy. The ideal AI would autonomously discern valuable information for its 'brain' versus information to be kept in external notes. This involves considering data saliency, frequency of repetition, and the utility of knowing a fact versus looking it up. The holy grail is for the model itself to learn these distinctions without explicit human guidance on what goes where, though user feedback would still be incorporated to personalize the learning process.
Efficiency as a metric of intelligence
Biderman firmly believes that efficiency and intelligence are inseparable. He criticizes the current dominant paradigm of 'doing more with more' in AI development, arguing that the next evolution involves 'doing more with less.' This efficiency is not merely about cost savings but about enabling more ambitious and complex tasks. By using fewer resources—tokens, compute, memory—AI can tackle longer-horizon problems and achieve breakthroughs that were previously impossible. Engram's work on token efficiency and compressed knowledge representations is a direct embodiment of this principle, aiming to build smarter, more capable AI systems that operate with greater resourcefulness.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Common Questions
Engram is an AI company founded by Dan Biderman that focuses on solving the 'AI memory problem.' They are developing methods to make AI models more efficient and accurate when dealing with vast amounts of data, going beyond simple context windows to create more intuitive and adaptable AI.
Topics
Mentioned in this video
Dan Biderman's AI company focused on solving the AI memory problem by developing efficient ways to interface with large corpora of knowledge.
Parameter-efficient fine-tuning method mentioned as part of Engram's long-term ambition for personalized AI models.
Company led by Assaf Rapaort, with connections to Israeli special forces and mentorship for Dan Biderman.
Company where Cade Daniel was an inference lead, highlighting his expertise relevant to Engram's infrastructure needs.
Co-founder and CEO of Engram, leading the discussion on AI memory and continual learning, while also cooking meatballs.
A professor at Stanford with whom Dan Biderman worked and who is a co-founder of Engram.
A professor at Stanford and co-founder of Engram, working on similar topics as Dan Biderman.
CEO of WHIS, who is a mentor to Dan Biderman and has a connection to Israeli special forces.
Mentioned as the subject of a Wikipedia article used as an example to illustrate the memory inefficiency of LLMs.
Former inference lead at Databricks and core contributor to VLM, now part of Engram's team.
Dan Biderman's hometown, where he visited his parents and discovered the Mediterranean meatball recipe.
University where current employee Drew is from, contributing to the diverse expertise within Engram.
Dan Biderman's home in San Francisco, where he challenges other AI leaders to come and cook with him.
A catalyst for Dan Biderman's deeper dive into the LLM world after seeing its capabilities.
A company where Dan Biderman worked on LoRA and met key people before co-founding Engram.
Mentioned as an example of large file systems in enterprise contexts that Engram works with.
Mentioned as an example of a powerful model that might be overkill for simple tasks but necessary for complex ones, relating to model routing.
Mentioned as a project where Cade Daniel was a core contributor, relating to inference leads.
More from Latent Space
View all 237 summaries
49 minWhy AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab
29 minPodcast Crossover: AIE, AGI, frontier lab strategy with @matthew_berman and @swyxtv
60 minThe 100,000 Sandbox Problem — Akshat Bubna, Modal CTO
109 min🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free