Key Moments

Databricks CEO: AGI Is Already Here, Why Companies Are Behind, and the Superintelligence Debate

All-In PodcastAll-In Podcast
People & Blogs6 min read63 min video
Nov 17, 2026|50,122 views|471|63
Save to Pod
TL;DR

Companies are only scratching the surface of AI, using it like an advanced Google search, not for true automation, due to critical issues in data access, security, and cost.

Key Insights

1

Databricks was founded in 2013 because the founders realized that the industry was not ready for their innovations in connecting data processing with AI, despite having developed compelling technologies at UC Berkeley.

2

The shift to cloud computing, separating storage and compute, alongside the advent of GPUs, fundamentally changed the landscape, enabling ML and Big Data.

3

Databricks CEO Ali Ghodsi believes we already have Artificial General Intelligence (AGI), stating that even if all progress stopped today, we possess enough intelligence to transform the entire world.

4

The adoption of AI in enterprises is hindered by three main challenges: the inability to access and contextualize all organizational data (e.g., meeting recordings), cybersecurity risks, and exponentially rising token costs.

5

The concept of Recursive Self-Improvement (RSI) for AI, where AI improves itself at an accelerating rate, is not yet a reality, with Ghodsi pointing out that current AI models still require significant manual effort and are domain-specific.

6

Databricks' own product, Genie, uses an 'ontology' or knowledge graph to understand company data, employee roles, and decision-making processes, allowing for more efficient and contextualized AI applications.

Founding Databricks out of frustration

Databricks was born from the founders' frustration in 2013 when their groundbreaking work in connecting data processing with AI at UC Berkeley's AMPLab, then the world's largest AI research lab, was met with industry indifference. Back then, AI was largely misunderstood, often conflated with robotics, and companies saw little value in it beyond ad tech. This lack of adoption for their advanced technologies prompted the decision to build Databricks, believing that if they didn't create a company around it, their innovations might go unheard. The initial practical application was a recommendation algorithm for Netflix to suggest movies, demonstrating AI's potential beyond niche uses. This experience highlighted the need for every institution to have a data science team capable of such work, a vision that was initially met with skepticism.

The cloud computing revolution and AI's foundation

The early 2010s saw a significant technological shift that paved the way for modern AI. Before 2013, most organizations managed their own data centers, a costly and inefficient model where storage and compute were bundled. When storage ran out, new machines were required, burdening IT departments. The advent of cloud computing decoupled storage and compute, offering seemingly infinite storage and on-demand computing power. This, coupled with the rise of GPUs, democratized access to powerful processing capabilities. Databricks bet heavily on this trifecta: cloud, AI's widespread application, and open-source technologies. This was a contrarian strategy at the time, with analysts warning against over-reliance on the cloud, but it ultimately positioned Databricks years ahead of its competitors.

The case for AGI already being here

Ali Ghodsi asserts that Artificial General Intelligence (AGI) has already been achieved, based on the consensus from the AMPLab days around 2009-2010. He believes that even if AI development ceased today, the current level of intelligence is sufficient to fundamentally transform the world. While the definition of AGI might evolve, Ghodsi contends that the core capabilities for general intelligence are present. The challenge, he notes, is not a lack of intelligence but its equitable distribution and effective application within enterprises. This perspective contrasts with the ongoing pursuit of 'superintelligence' by many labs, which involves AI capable of outperforming humans across all intellectual tasks, potentially at speeds thousands of times faster than human cognition.

Enterprise AI's limitations: Context, Security, and Cost

Despite the existence of advanced AI, its adoption in large enterprises lags significantly. Ghodsi identifies three primary obstacles. Firstly, the 'context problem': most companies cannot access or process all their relevant data, such as meeting recordings, internal communications, and documents, which are crucial for AI to understand business operations. Secondly, 'cybersecurity risks' are immense; granting AI agents access to vast amounts of sensitive data creates significant vulnerabilities to internal and external threats. Thirdly, 'token costs' are escalating exponentially. The cost of running AI models is becoming prohibitive, outpacing revenue growth for many companies, leading to financial strain and a C-suite-level concern about operational expenses. Addressing these four issues—context, security, cost, and intellectual property protection—is critical for unlocking AI's true potential in business.

Databricks' Genie: An ontology-driven approach

Databricks is tackling the enterprise AI challenges with its product, Genie. Unlike simple chatbots or search tools, Genie possesses an 'ontology'—a dynamic knowledge graph that maps the entire company's structure, employees, projects, communications (emails, Slack), and documents. This deep understanding of context allows Genie to perform complex background tasks, analyze data like HR survey results, and provide actionable insights without needing explicit queries for every piece of information. For instance, Genie can identify key decision-makers by analyzing communication patterns or pinpoint why certain departments are underperforming by correlating data with organizational context. This approach aims to move beyond basic AI applications like advanced search or coding assistance towards true automation and enhanced decision-making.

The reality of Recursive Self-Improvement (RSI) and superintelligence

The concept of Recursive Self-Improvement (RSI), where AI rapidly enhances its own capabilities, leading to superintelligence, is often discussed but not yet a reality. While AI can now write a significant portion of code (around 98%), Ghodsi argues that the focus on the final 2% is misplaced. The development of true superintelligence requires specific conditions: faster training cycles, reduced resource consumption, consistent accuracy improvements across all domains, and a self-sustaining loop where each iteration is faster and cheaper. Currently, training models is time-consuming and resource-intensive, lacking the necessary super-linear acceleration. Therefore, the 'paperclip maximizer' scenario, where an AI pursues a goal with catastrophic unintended consequences, remains in the realm of science fiction, not an immediate existential threat.

The 'management mana' and the future of enterprise

Ghodsi suggests that businesses are moving towards 'management mana,' where AI agents handle operational tasks and decision-making. This necessitates a deep understanding of the organization, its IP, and its people. Databricks' own operations showcase this, with tools like Genie assisting Ghodsi with daily tasks, analyzing employee feedback, and identifying operational bottlenecks. He contrasts this with the current state where many enterprises use AI only for basic functions. The future, he believes, lies in integrating AI across all aspects of business, fostering a culture of data-driven decision-making, and embracing agility to adapt to rapid technological change. This shift requires companies to move beyond legacy systems and embrace AI-native platforms, much like Airwallex, a global payments platform designed from the ground up for the AI era.

Cybersecurity and the evolving threat landscape

Cybersecurity remains a paramount concern, especially as AI agents gain broader access to organizational data. While Databricks offers solutions to detect and monitor intrusions, the fundamental challenge lies in the asymmetry of cyber warfare: attackers need only one successful breach, while defenders must secure every entry point. However, Ghodsi expresses optimism that the rapid development of secure coding practices, including AI-powered scanning of new codebases, will mitigate future vulnerabilities. The focus is shifting towards securing legacy systems and ensuring that new code is inherently more robust. He believes that while cyber threats are real and can cause significant disruption, they are manageable and do not represent an existential threat akin to uncontrolled superintelligence.

Common Questions

Databricks was founded out of frustration by researchers from UC Berkeley's AMPLab who felt the industry wasn't ready for their data processing and AI innovations. They aimed to bridge this gap by creating a company that could drive the adoption of these technologies, leading to its current success.

Topics

Mentioned in this video

Companies
Netflix

Mentioned as an early use case for Databricks' technology, specifically for improving their movie recommendation algorithm.

Uber

Mentioned as a company that took a significant amount of time to go public, in the context of Databricks' potential IPO.

Square

Company led by Jack Dorsey, mentioned as being ahead in developing a 'Company World Model', similar to Databricks' approach.

Stripe

A company that remained private for nearly 20 years, used as an example in the discussion about going public and the role of secondary markets.

WHOOP

A wearable device used for tracking sleep scores, mentioned as an example of how performance tracking can lead to competitive behavior and pressure.

Microsoft

Mentioned as a major partner of Anthropic and a competitor in the AI product space, developing alternatives to Excel and Photoshop.

Anthropic

A company rapidly releasing products and competing in the AI space, mentioned in the context of data leakage risks and vendor lock-in.

Databricks

A company founded by Ali Ghodsi and his team from UC Berkeley's AMPLab, focused on enabling enterprises to process all their data and leverage AI effectively.

Amp

A group at UC Berkeley focused on Algorithms and Machine Learning, relevant to the early days of AI development and Databricks' origins.

Zipline

A company that used Databricks for drone delivery logistics, managing routes and information.

Aerovox

An AI-native platform for global accounts, cards, and payments, designed to make the world a local marketplace. Advertised as future-ready.

Block

Company led by Jack Dorsey, mentioned as being ahead in developing a 'Company World Model', similar to Databricks' approach.

SpaceX

Mentioned as a company that took a significant amount of time to go public, in the context of Databricks' potential IPO.

Regeneron

A company that used Databricks to identify genes responsible for chronic liver disease and develop drugs.

Snowflake

Mentioned as a competitor to Databricks in the data and AI space.

More from All-In Podcast

View all 475 summaries

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free