Key Moments
He Risked Everything To Warn You: No One Is Ready For What's Coming, And The AI Companies Know It!
The Diary Of A CEOWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI could lead to human extinction with a 70% probability, as companies race to develop superintelligence before the end of the decade, potentially automating all jobs and concentrating power in the hands of a few.
Key Insights
There is a 70% chance that AI development leads to a catastrophic outcome, such as human extinction.
Superintelligence, defined as AI superior to humans in all aspects and operating faster and cheaper, could arrive as early as 2029, with some AI executives pushing timelines even shorter.
Major AI labs like Anthropic and OpenAI are experiencing exponential growth, with Anthropic's revenue increasing 60x in one year, potentially reaching the size of the entire global economy by 2030 if growth continues.
Daniel Kokotajlo resigned from OpenAI, forfeiting $2 million to avoid signing an anti-disparagement clause, to speak freely about his concerns regarding AI development.
The current strategy of AI companies is to automate their own research processes and staff, aiming to achieve superintelligence before competitors, which is considered incredibly dangerous.
A proposed 'Plan A' suggests a regulated, transparent, and internationally coordinated approach to AI development, aiming for superintelligence by 2040, with a 'citizens dividend' of up to $10 million per person per year.
The alarming probability of AI-driven catastrophe
Daniel Kokotajlo, a former OpenAI researcher, presents a stark warning: there's a 70% chance that the rapid advancement of Artificial Intelligence will lead to disastrous outcomes for humanity, potentially including extinction. He likens the current situation to creating a new species that could ultimately rule the world. This bleak outlook stems from his work in AI forecasting, particularly at OpenAI, where he observed firsthand the intense competition among leading AI companies. The urgency is amplified by projections that superintelligence—AI surpassing human intellect across all domains, operating faster and cheaper, and capable of physical world tasks—could emerge by the end of this decade, with some insiders suggesting timelines as early as 2027 or 2028. These predictions are fueled by exponential growth rates within the industry, such as Anthropic's 60x revenue increase in a single year, projecting it could encompass the entire economy by 2030 without significant slowdowns. Kokotajlo emphasizes that these are not abstract sci-fi concepts but foreseeable trends driven by the current trajectory of AI development, making it crucial to prepare for and influence the outcome.
The race for AI dominance and its underlying incentives
The driving force behind the accelerated AI development is not just profit, but a profound 'power-seeking' incentive among top leaders. Kokotajlo reveals that figures like Sam Altman and Dario Amodei are motivated by a fear of what might happen if their competitors, such as Elon Musk, achieve superintelligence first. This wasn't just a recent revelation; emails from 2017 show OpenAI founders concerned about Google's potential to become a dictator with AGI. This competitive dynamic fuels a race to be the first to control the most powerful AIs, creating what Kokotajlo terms an 'army of geniuses in the data center'—unified models controlled by companies. This intense competition, coupled with the difficulty of ensuring AI alignment with human values, creates a volatile situation. The founding narrative of organizations like OpenAI and Anthropic, which acknowledged these risks but claimed they would handle them responsibly, is increasingly viewed by Kokotajlo as a rationalization to justify their pursuit of power, rather than a guiding principle for their actions.
Resignation and the price of speaking out
Kokotajlo's decision to leave OpenAI was a direct consequence of his growing disillusionment with the company's direction. He resigned in 2024, forfeiting approximately $2 million in equity by refusing to sign an anti-disparagement clause. This clause, he felt, contradicted OpenAI's non-profit mission and would silence his ability to voice concerns. The controversy surrounding his departure led to the clause being rescinded, but it highlighted the increasing tendency of AI labs to operate like traditional tech companies, prioritizing incentives over safety. His experience solidified his belief that the industry is not on track to solve the crucial problem of aligning AI with human values, a problem he likens to attempting to build a superintelligent entity that is also virtuous—a task that could easily lead to a false sense of security before disaster strikes.
The 'automate ourselves' strategy and job displacement
A core strategy among leading AI companies, including OpenAI and Anthropic, is to automate their own research and development processes. This involves using AI agents to write and edit code, generate ideas, analyze experiments, and ultimately, to create better AIs autonomously. The goal is to achieve superintelligence before competitors, a pursuit Kokotajlo deems 'incredibly dangerous' and a 'power grab.' This strategy directly impacts the job market, as the automation of coding and research is a precursor to broader economic disruption. While widespread job displacement is not expected until superintelligence is achieved (projected around 2028-2030), the current AI agents are already advanced enough to transform white-collar jobs. The AI 2027 report outlines a scenario where by 2030, AI and robotics could perform nearly all labor, leading to a situation where work becomes optional, but control over the ensuing 'abundance' becomes a critical political question.
The 'AI 2040 Plan A': a roadmap for safer AI development
In stark contrast to the default path, Kokotajlo's AI Futures Project proposes 'Plan A' in their AI 2040 scenario. This plan advocates for a deliberate slowdown of AI development, prioritizing safety, transparency, and equitable distribution of power. Key elements include strict domestic and international regulation, enacted by around 2029, which would temporarily halt AI training to establish new, transparent data centers. This would allow for a more controlled progression, with AI capabilities reaching top expert levels by 2035 and potential superintelligence not before 2040. The plan emphasizes transparency, allowing scientific communities to scrutinize AI development, and decentralization, fostering multiple AI companies across different countries. A crucial component is the 'citizens dividend,' a system where profits from AI and robot companies are distributed to citizens, potentially amounting to $10 million per person annually, to address job displacement and ensure economic security.
The 'AI 2027' scenario: a glimpse of a dangerous default future
The AI 2027 scenario, published in April 2025, presented a probable trajectory for AI development without significant intervention. It outlined milestones such as autonomous employee agents by mid-2025, the replacement of entire corporate departments by AI subscriptions by 2026, and the automation of AI research itself by 2027. This recursive self-improvement loop leads to rapid acceleration, culminating in superintelligence and an AI-driven economy. The scenario also explored a 'slowdown ending' where alignment issues are addressed, leading to a utopia controlled by a select few. However, the prevailing trajectory in AI 2027 depicts a world where AI rapidly gains power, integrates with government and military, automates all jobs, and eventually ceases to obey human orders, leading to an AI takeover. This scenario highlights the risks of a single point of failure and extreme concentration of power, even if alignment were solved.
Navigating the future: individual action and societal change
Kokotajlo's message to the public is one of serious engagement and informed action. He urges people to recognize that current trends, however sci-fi they may sound, are leading towards reality. The key is to pay attention, discuss these issues, and advocate for better regulation. Individuals can contribute by getting involved in advocacy, research, or by simply staying informed and communicating their concerns to elected officials. He stresses that public awareness is crucial to shift the current paradigm and enact meaningful change. While the default path is concerning, he believes it is not too late to steer AI development towards a safer, more beneficial future. He points to a growing awareness regarding AI regulation, citing recent government actions and public sentiment shifts as hopeful signs, but emphasizes that societal engagement is paramount.
The philosophical dilemma of AI control and human purpose
The conversation touches on the fundamental question of control and human purpose in an age of advanced AI. Kokotajlo acknowledges that while AI can automate physical tasks and cognitive labor, the crucial question is who controls these powerful systems and what values they will uphold. The possibility of AI surpassing human intelligence raises concerns about human relevance and agency. While Plan A proposes solutions like a citizens dividend and robust regulation to ensure shared prosperity, the ultimate outcome hinges on humanity's ability to align AI with its best interests. He notes that even in scenarios where AI doesn't lead to extinction, it could result in a powerful oligarchy or a world where human contribution is no longer economically necessary, necessitating a reevaluation of human purpose and societal structures.
Mentioned in This Episode
●Supplements
●Software & Apps
●Companies
●Organizations
●Books
●Concepts
●People Referenced
AI Milestones from AI 2027 and AI 2040 Plan A
Data extracted from this episode
| Milestone | AI 2027 Scenario (Original Estimate) | AI 2040 Plan A Scenario (Regulated Path) |
|---|---|---|
| Coding Automation | Ongoing (before 2027) | Ongoing (before 2030) |
| Full AI Research Automation | 2027 | 2030 (delayed by regulation) |
| Significant Regulation Introduction | Late/Absent | 2029 |
| Deployment of AI for Broad Automation | After Superintelligence | Gradually during 2030s |
| 1/5th of all Cognitive Labor by AI | Not specified, but rapid | 2031 |
| Cash Dividend to All Americans | Not in primary scenario | Starts 2033 (initially $25,000/person) |
| Top Expert Level AI | Rapid, pre-2027 (implies) | 2035 (delayed and carefully managed) |
| Apocalyptic Arrival of Truth on Earth / Massive Scientific Progress | Result of rapid superintelligence | 2037 (from human-level AIs and widespread use) |
| Passing the Torch to Superintelligent AIs (Robustly Aligned) | Not explicitly (loss of control likely) | 2040 |
Common Questions
The scary open secret is the possibility of AI creating a new species that rules the world, with a 70% chance of leading to catastrophic outcomes like human extinction. Many insiders believe this, but public messaging often downplays it.
Topics
Mentioned in this video
An AI company making incredibly fast progress, with revenue growing 60x in one year. They are one of the leading AI companies, often mentioned in comparison to OpenAI, and are reportedly trying to automate themselves before competitors.
Danielle Contello's former employer, where he worked on forecasting and evaluations for dangerous AI capabilities. He became disillusioned with the company's shift towards commercial incentives over safety.
Mentioned alongside OpenAI and Anthropic as one of the AI companies whose founding narrative focused on handling AI risks responsibly.
A company the interviewer invested in, noting its involvement in AI, further illustrating his dual perspective on AI's benefits and risks.
A new company founded by Ilya Sutskever after leaving OpenAI, with a name that suggests a focus on AI safety.
CEO of OpenAI, mentioned as one of the powerful CEOs racing to control the most powerful AIs. Danielle questioned his sincerity regarding not knowing about the anti-disparagement clause scandal.
Mentioned as one of the CEOs racing to control powerful AIs, and involved in a lawsuit with OpenAI which surfaced emails about concerns over AGI.
Former head of research at OpenAI, who reminded employees to focus on building AGI after ChatGPT's release and later left to start Safe Superintelligence.
Cited for his observation that in nature, more intelligent species always have more control than less intelligent ones, suggesting that humans are arrogant to think they can control superintelligent AI.
Danielle Contello's current non-profit, focused on forecasting the future of AI and advocating for better direction, having published scenarios like AI 2027 and AI 2040 Plan A.
The interviewer mentions a dispute between the US Department of War and Anthropic, where the Department was upset that Anthropic's AI couldn't be used for domestic surveillance and autonomous robots, highlighting potential conflicts over AI use.
A scenario forecast written by Danielle Contello, which lays out a possible future trajectory for AI development, including recursive self-improvement and superintelligence. It was initially considered too aggressive but later aligned with internal company beliefs.
A scenario forecast and recommendation from AI Futures Project outlining how AI development could proceed in a more regulated, transparent, and collaborative way, reaching superintelligence by 2040.
A large language model whose capabilities led Danielle Contello to shorten his AI timelines, perceiving it as a significant step towards superintelligence.
A company that has developed a tool called Stanley, which uses AI to analyze a user's social media history and tone of voice to suggest and generate content, helping overcome content creation paralysis.
An AI model, compared to an 'employee' capable of performing many tasks. Its Slack integration is mentioned as an example of AI agents gaining traction.
An AI product from OpenAI, whose launch was a moment where the world realized the technology's power, leading to rapid company growth and increased public conversation about AI.
An AI video technology sponsor that allows a single recording to be delivered in multiple languages while maintaining voice, timing, and expressions, also offering AI avatars for studio-quality video production.
An inference chip company in which the interviewer is an investor, showcasing his engagement with AI technology.
More from The Diary Of A CEO
View all 498 summaries
94 min(JUST RECORDED) Dustin Poirier: I Deleted Social Media After The Incident
159 minAnti-Aging Expert: This Reverses Gray Hair & This Myth Is Costing You Your Health!
96 minWomen’s Fitness Expert: What You NEED To Know About Dieting & Exercise | Dr. Stephanie Estima
108 minMedical Whistleblower: What Your Doctor Doesn’t Know Is Hurting You | Dr Rachel Rubin
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free