Key Moments
How Exa is Building the Perfect Search Engine | Deep Dives with a16z
a16zWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Exa's CEO claims their search engine is superior to Google's for AI agents, offering comprehensive results but facing extreme quality demands.
Key Insights
Exa aims to build a perfect search engine, believing that improving search has vast downstream implications across all industries and aspects of human life.
Traditional search engines like Google are optimized for human clicks and quick answers, failing to provide the depth and comprehensiveness required by AI agents.
AI agents require different search capabilities than humans, needing complex query handling, high comprehensiveness (thousands or millions of results), and extreme customizability.
Building a search engine for AI agents is easier because human click data is less relevant, and LLMs simplify tasks like re-ranking; however, it's harder due to the demand for much higher quality (99.999% reliability).
Exa positions search as a fundamental problem that underlies many complex issues, including political polarization and loneliness, arguing that better information access can lead to more reasonable outcomes.
The 'token apocalypse' can be mitigated by using retrieval to make smaller models more efficient and accurate, potentially saving customers 20x on costs compared to other providers.
The pursuit of perfect search as a life mission
Will Bryk, co-founder and CEO of Exa, views building a better search engine as a lifelong mission. His interest began in childhood, driven by a desire for higher quality knowledge. This evolved into attempts to create better news organizations and search engines throughout college. The advent of powerful transformers in 2021 made the idea of building a search engine superior to Google a tangible possibility. Bryk believes that a perfect search engine would have profound positive implications for the world, impacting every industry and human endeavor. This conviction fueled the creation of Exa, aiming to achieve what he considers a critical but unpursued opportunity.
Limitations of traditional search for AI agents
While Google excels at surface-level information and quick answers for billions of human users, it falls short for deeper queries and, crucially, for AI agents. Google's optimization for human clicks and understanding ambiguous, tired queries (e.g., from billions of similar human searches) doesn't translate to the needs of autonomous systems. Bryk highlights that businesses and agents have complex needs, such as finding every competitor for a company or identifying specific types of job candidates, which Google is not built to handle. The recent evolution of Google Search with AI modes still focuses on consumer use cases, leaving a significant gap for deep, complex queries that Exa aims to fill.
Designing search for the agentic economy
The search landscape shifts dramatically when the user is an AI agent rather than a human. Agents, unlike humans who are constrained by time, can process information rapidly and have infinite patience for complex queries. They require comprehensiveness, wanting 'everything' rather than just a curated list of 10 results. For instance, an investor agent needs complete information on biotech companies to make crucial decisions without missing any vital data points. Exa's search engine is designed to handle complex semantic and keyword queries, offering high customizability through various filters and toggles. This allows agents to refine their searches iteratively until they achieve their desired outcome, unlike traditional search where users often have to rephrase queries.
Easier yet harder: The dual nature of building for agents
Bryk explains that building a search engine for agents is both easier and harder than for humans. It's easier because the vast troves of human click data that Google relies on are far less critical for agents, opening up a new playing field. Tasks like re-ranking, which previously required large teams, can now be handled much more efficiently with LLMs and a smaller engineering focus. However, it's harder because the demands for quality and reliability are exponentially higher. Agents and businesses need near-perfect results (99.999% quality), pushing Exa towards extreme precision, similar to how LLM advancements are measured in 'extra nines' of performance. This constant push for perfection from demanding business use cases is what drives Exa's development.
Search as a foundational problem for complex challenges
Exa views search not just as an information retrieval problem but as a core solution to many broader societal and business challenges. Bryk argues that issues like political polarization stem from people receiving misleading or inaccurate information, and a perfect search engine providing accurate, controllable, and comprehensive data could help foster more reasonable discourse. Similarly, loneliness can be seen as a search problem, where individuals struggle to find others with shared interests. Exa's focus on company and people search for go-to-market intelligence, and its ambition to provide perfect retrieval, aims to address these fundamental needs by organizing the world's information more effectively.
Data acquisition and incentivizing content providers
The web is becoming increasingly closed, raising concerns for data providers worried about being 'Stack Overflowed.' Exa aims to create a system where content providers benefit financially from the emerging agentic economy. Bryk envisions a future where a significant portion of the value generated by AI agents is distributed back to content creators, potentially creating a more favorable economic model than the current ad-driven internet. This involves accumulating diverse data, including web content and potentially data not yet recorded, and developing powerful retrieval models to access it effectively.
Enhancing coding agents and mitigating token costs
Coding agents, like all agents, require up-to-date technical documentation and inspiration. Exa's search capabilities are crucial for providing the freshest information to prevent errors in code generated by these agents, where retrieval quality directly impacts code accuracy. Furthermore, Exa plays a role in solving the 'token apocalypse' by enabling retrieval-augmented generation. This allows smaller, more efficient models to act like larger ones by accessing relevant information, drastically reducing token consumption and computational costs—potentially saving customers up to 20x. This approach aligns with the trend of using smaller, specialized models integrated with retrieval tools.
Research-driven innovation and the future of search
Exa operates like a research lab, applying LLM training techniques like pre-training, post-training, and reinforcement learning (RL) to search models. Their research shows that RL on Exa yields better results with fewer calls compared to other search tools, highlighting Exa's design for agents. Bryk believes the industry faces infrastructure and data bottlenecks, with a future need to unearth and index vastly more data beyond the current web. He predicts agentic search will become a larger business than Google search by the 2030s, driven by an exponential increase in agent-driven information requests that will permeate society like electricity. The company's culture emphasizes passion, fun, and empowering individuals to work on projects that excite them, attracting top talent by offering challenging, mission-driven work.
Mentioned in This Episode
●Software & Apps
●Companies
●Concepts
●People Referenced
Common Questions
Exa is designed for deep, complex queries and AI agents, aiming for comprehensive information retrieval rather than just quick answers for billions of consumers. It focuses on structured data and capabilities that Google currently lacks.
Topics
Mentioned in this video
The dominant search engine company, often seen as a monopoly, with limitations in deep search capabilities and specific B2B/agent use cases.
A company associated with the 'Tinker' tool used by Exa.
While not explicitly named, the mention of ChatGPT and large language models implies the advancements made by companies like OpenAI.
A company whose AI coding agents, like Devon, use Exa's search capabilities.
Company founded by Elon Musk, where Will had a magical internship experience, witnessing the first landing on the barge.
A large language model released by OpenAI, which emerged shortly after Exa's initial public release, highlighting a new wave of information finding.
A database mentioned as an example of a development tool that agents might select, indicating a paradigm shift where agents, not humans, decide winning tools.
An early language model mentioned in a thought experiment about the cost of building a better search engine than Google.
An AI coding agent that uses Exa, showing improved accuracy and fewer mistakes when powered by Exa's search.
A type of database that Exa uses and that the company's founder helped build, with research into high-throughput solutions to handle increased search demand.
A search engine company aiming to build a 'perfect search engine' that goes beyond Google's capabilities, particularly for complex queries and AI agents.
A database mentioned as an example of a development tool that agents might select, indicating a paradigm shift where agents, not humans, decide winning tools.
A tool used by Exa for research, with a shoutout to Thinking Machines.
Mentioned as a family friend who recommended Exa for web access.
A prominent figure in AI, he retweeted Exa's initial release and later tweeted about the trend towards smaller models using tools, which is highly relevant to Exa's strategy.
Founder of SpaceX, mentioned for his leadership style, detail-orientation, and ability to create inspiring, mimetic mission names.
Author of the article 'Grind Slop,' which critiques the glorification of intense work and the squandering of economic surplus.
Mentioned as an example of a topic where deep, nuanced understanding is difficult to achieve with current search engines like Google, highlighting Exa's potential.
Mentioned as the location where the speaker wishes they could find 'flying car enthusiasts' via a perfect search engine.
More from a16z Deep Dives
View all 55 summaries
51 minWhy AI Agents Need Context | Deep Dives with a16z
46 minPyramid of Work and The Future of Enterprise Automation | The a16z Show
37 minRebuilding Global Banking With Stablecoins and AI | Deep Dives with Jeeves CEO
49 minHow AI Is Entering the Physical World | Deep Dives with a16z
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free