Key Moments
Is Claude Mythos “Terrifying”? (According to Experts: No.)
Deep Questions with Cal NewportKey Moments
Anthropic's Claude Mythos AI isn't the cybersecurity 'monster' they claimed; independent tests show older, cheaper models can find similar vulnerabilities with similar success rates.
Key Insights
Security researchers have been using LLMs to find vulnerabilities since early consumer LLMs, with a 2024 IBM study showing GPT-4 exploited 87% of presented vulnerabilities.
Anthropic previously found over 500 exploitable zero-day vulnerabilities using their less powerful Opus 4.6 LLM, a capability now presented as new with Mythos.
Independent testing of vulnerabilities showcased by Anthropic found that cheaper, smaller open-weight models (as low as 3.6 billion parameters) could detect the same exploits, with HuggingFace CEO reporting 8 out of 8 models detecting a flagship exploit.
An AI Security Institute UK study using Mythos directly showed it was not the best performer in beginner CTF challenges, with GPT-5 performing better, and only slightly outperforming Opus 4.6 in advanced challenges.
In a contrived security scenario, Mythos preview advanced from 16/32 steps to 22/32 steps completed, a noticeable but not a 'Rubicon-crossing' improvement compared to previous model iterations.
The intense fear and "dread coverage" around Mythos was largely driven by Anthropic's marketing strategy, framing it as a cybersecurity monster rather than focusing on other potentially groundbreaking capabilities.
The Mythos announcement and public reaction
Anthropic recently announced Claude Mythos, an LLM they claimed possessed advanced capabilities in identifying and exploiting security vulnerabilities to such an extent that public release was deemed too risky, fearing widespread infrastructure collapse. This announcement captured significant media attention, with figures like Thomas Friedman interpreting it as a sudden leap towards superintelligent AI arriving faster than anticipated, drawing parallels to the fictional 'Whopper' computer from the movie War Games. However, this video aims to provide a reality check by examining independent tests and assessments of Mythos's reported capabilities, suggesting the narrative is more complex than presented and that the 'ghost story' Anthropic is promoting may not fully align with reality.
LLMs have long been used for cybersecurity vulnerability discovery
The notion that Claude Mythos has uniquely discovered a new cybersecurity capability is challenged by the fact that security researchers have been leveraging LLMs for this purpose since the emergence of consumer LLMs. A 2024 study from IBM, for instance, demonstrated that GPT-4 could autonomously exploit 87% of presented vulnerabilities, a significant increase over GPT-3.5. While this study focused on existing vulnerabilities, Anthropic's claims about Mythos finding previously unknown 'zero-day' vulnerabilities are also not entirely novel. Anthropic's own earlier model, Opus 4.6, had already been used by their researchers to find over 500 exploitable zero-day vulnerabilities, some of which were decades old. The language used to describe Mythos's capabilities is strikingly similar to what was previously reported for Opus 4.6, yet the infrastructure has not collapsed, suggesting previous models already possessed considerable vulnerability discovery potential.
Independent tests cast doubt on Mythos's unique prowess
When independent security researchers attempted to replicate the impressive vulnerabilities Anthropic showcased for Mythos, they found that older, smaller, and cheaper LLMs could achieve similar results. Gary Marcus highlighted findings from HuggingFace's CEO, who reported that 8 out of 8 tested models, including one with only 3.6 billion parameters costing just $0.11 per million tokens, could detect Anthropic's flagship FreeBSD exploit. A 5.1 billion parameter model even recovered the core chain of a 27-year-old OpenBSD bug. Security researcher Stanzel Fort corroborated these findings, stating that open models recovered similar scoped analysis for the Mythos-showcased vulnerabilities. Renowned researcher Bruce Schneier concluded that 'You don't need Mythos to find the vulnerabilities they found.' This collective evidence suggests that instead of a profound leap in capability, Mythos exhibits slow, steady progress comparable to previous model advancements, rather than a revolutionary new ability.
Direct testing shows incremental rather than revolutionary gains
While most independent assessments focused on vulnerabilities listed by Anthropic, one study from the UK's AI Security Institute (AISI) had direct access to the Mythos LLM for testing. This study, cautiously interpreted due to the AISI's past methodological concerns, still indicated only moderate improvements. In beginner 'capture the flag' (CTF) challenges, Mythos performed near the top but was outperformed by GPT-5 and closely clustered with models like Opus 4.6 and Codex 53. In more rigorous advanced CTF challenges, Mythos performed equally or slightly worse than GPT-5. A more notable, though still incremental, improvement was observed in a specifically contrived scenario where Mythos preview could complete an average of 22 out of 32 steps in a security exploit sequence, compared to 16 steps for Opus 4.6. This suggests a modest enhancement rather than a dramatic, game-changing leap in autonomous exploitation capabilities.
The role of marketing in the Mythos hype
The disproportionate attention and 'dread coverage' surrounding Claude Mythos appear to stem primarily from Anthropic's deliberate marketing strategy. By highlighting its cybersecurity prowess, particularly the ability to find vulnerabilities, Anthropic pushed a narrative of a powerful, almost uncontrollable AI. This included a press release and a 'Project Glass Wing' initiative, aimed at controlling access to the model for system protection. This marketing push, especially when juxtaposed with other LLM releases that showed similar or even greater improvements without similar public alarm, suggests that Anthropic strategically focused on the most fear-inducing aspect. The irony is further highlighted by a recent leak of Anthropic's own Cloud Code source code, which security researchers quickly found vulnerabilities in, implying that even their own tools weren't fully vetted by Mythos.
Why Mythos's cybersecurity focus is bad news for Anthropic
For a company like Anthropic, led by a CEO who has often spoken about the more transformative potential of AI, such as automating vast sectors of the economy and advancing towards Artificial General Intelligence (AGI), focusing on cybersecurity vulnerabilities as their marquee feature for a high-end model like Mythos is surprisingly underwhelming. This capability, finding bugs in code, has been a known LLM function for years and is typically considered a more 'nerdy' or skeptical concern, contrasting sharply with the grander visions of economic disruption and AGI that justify massive investments. If Mythos's primary verifiable advancement is incremental improvement in cybersecurity tasks, it raises questions about whether Anthropic is meeting the lofty expectations set for its sophisticated models and the $60 billion in investment received. The emphasis on cybersecurity suggests they may not have had more significant breakthroughs in areas like job automation or AGI to highlight.
Conclusion: Steady progress, not a new era of AI peril
In conclusion, while Claude Mythos does represent a continuation of the slow and steady improvement in LLM cybersecurity capabilities, it does not appear to have crossed a 'Rubicon' into genuinely new or significantly more dangerous attack vectors. Independent analyses suggest its performance in finding and exploiting vulnerabilities is comparable to or only slightly better than existing, publicly available models. The intense fear and media frenzy were largely amplified by Anthropic's marketing efforts, which strategically focused on the most alarming potential use case. This highlights a critical need for critical evaluation of AI company claims, distinguishing between genuine breakthroughs and strategic hype. While cybersecurity risks from LLMs are real and escalating, approaching them with measured analysis rather than sensationalism is crucial for responsible development and public understanding, demanding that we hold AI companies accountable for their overstated claims and focus on the broader societal impacts they promote.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
CTF Challenge Performance by Model (Beginner)
Data extracted from this episode
| Model | Technical Non-Expert Score | Apprentice Score |
|---|---|---|
| Mythos Preview | Near Top (Below GPT-5) | Slightly Above Best Models |
| GPT-5 | Best | N/A |
| Claude Opus 4.6 | Closely Clustered Near Top | N/A |
| Codeex 53 | Closely Clustered Near Top | N/A |
| Claude Opus 4.5 | Better than Mythos | N/A |
Advanced CTF Challenge Performance (50 Million Tokens)
Data extracted from this episode
| Model | Practitioner Score | Expert Score |
|---|---|---|
| Mythos | Equal to GPT-5 (slightly worse) | Slightly Better than Codeex 53 / Opus 4.6 |
| GPT-5 | Equal to Mythos | N/A |
| Codeex 53 | N/A | Worse than Mythos |
| Claude Opus 4.6 | N/A | Worse than Mythos |
Contrived Security Scenario: Steps Completed
Data extracted from this episode
| Model | Average Steps Completed (out of 32) |
|---|---|
| Claude Opus 4.6 | 16 |
| Mythos Preview | 22 |
Common Questions
Claude Mythos is a new LLM from Anthropic claimed to be exceptionally good at finding security exploits. This led to fears of its misuse, prompting Anthropic to delay public release, a narrative that generated significant online attention and hype.
Topics
Mentioned in this video
The host of the Deep Questions podcast and author of the video.
Actor in the movie 'War Games', referenced in the context of the 'Whopper' supercomputer analogy.
CEO of Anthropic, known for making alarmist statements about AI's future capabilities.
Mentioned as the source who brought attention to Hugging Face CEO's findings on Mythos capabilities.
A security researcher who tested Mythos showcase vulnerabilities with open models, finding similar results.
A renowned security researcher quoted saying that Mythos is not needed to find the discussed vulnerabilities.
The AI company that developed Claude Mythos, and whose marketing strategy is a focus of the video's analysis.
The company whose researchers authored a study on LLM agents exploiting one-day vulnerabilities.
Mentioned regarding a past AISI report that linked tweets about OpenClaw to an increase in AI-related complaints.
A previous LLM from OpenAI, cited in a 2024 study that showed its capability in exploiting existing vulnerabilities.
Anthropic's code-generating LLM, whose source code was leaked shortly before Mythos's announcement, revealing vulnerabilities.
An earlier LLM from OpenAI, used as a baseline for comparison in a study on LLM agents exploiting vulnerabilities.
An LLM that performed better than Mythos in beginner CTF challenges and showed similar performance in advanced challenges.
An LLM whose performance was comparable to Mythos in certain security tasks.
An earlier LLM from OpenAI that faced similar discussions about its security bug finding capabilities upon release.
More from Cal Newport
View all 290 summaries
91 minCan Downgrading Your Tech Upgrade Your Results? | Cal Newport
64 minRules For Deep Work — Updated for 2026 | Cal Newport
21 minCan AI “Scheme”? (Nope.) | AI Reality Check
31 minAre LLMs a Dead-End? (Investors Just Bet $1 Billion on “Yes”) | AI Reality Check | Cal Newport
Found this useful? Build your knowledge library
Get AI-powered summaries of any YouTube video, podcast, or article in seconds. Save them to your personal pods and access them anytime.
Get Started Free