Key Moments
The Biggest AI Risks Ahead | Susan Schneider
Closer To TruthWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Mind uploading is conceptually flawed and likely centuries away, yet AI risks like deepfakes and autonomous weapons are imminent, demanding global cooperation.
Key Insights
The concept of 'uploading' one's brain to the cloud is not only technically challenging, likely centuries away, but also conceptually incoherent, as demonstrated by thought experiments involving duplicates or surviving originals.
Transhumanist visions of merging with AI to keep pace with superintelligence are rated a mere 10% likely, with the speaker believing AI will outthink humans, but not necessarily become conscious in a way we can merge with.
The likelihood of AI generating unpleasant surprises for humanity is extremely high, rated at 99%, due to increasing autonomy and the proliferation of capabilities like deepfakes and autonomous weapons.
Current advanced AI systems like LLMs are not considered conscious, but the possibility of AI consciousness in principle is rated 90% likely, particularly if based on biological or quantum-emulating systems.
The proliferation of AI capabilities, simpler to build than nuclear weapons, significantly increases the risk of rogue states, non-state actors, and malevolent individuals causing widespread destruction.
Global cooperation, international standards, and regulations are deemed essential to mitigate AI risks, as no single group can control these increasingly complex and emergent AI systems.
The conceptual pitfalls of mind uploading
Susan Schneider, author of "Artificial You" and director of the Center for the Future Mind, critically examines the transhumanist aspiration of uploading human consciousness to the cloud. She argues that this vision, often touted by techno-optimists, is not only technologically distant, likely centuries away, but also fundamentally lacks conceptual coherence. Schneider draws on philosophical thought experiments, like those posed by Derek Parfett, to illustrate the problem: if an upload process were non-destructive, one would exist simultaneously in their physical body and in the digital realm. This raises the question of which is the 'real' you, or if both are, leading to the possibility of infinite duplicates. This paradox suggests that consciousness transfer, as envisioned in uploading, is not a straightforward technological feat but a deeply philosophical and potentially incoherent one. The underlying assumption of computational functionalism—that consciousness can be instantiated in any material as long as it performs the right functions—is also questioned as a necessary but not sufficient condition for uploading.
Transhumanist ambitions and their low probability
The discussion delves into the transhumanist movement's goals of radically transforming the human condition through technology, including longevity and brain augmentation. While some proponents, like Elon Musk, suggest merging with AI via brain chips to keep pace with superintelligence, Schneider expresses significant skepticism. She rates the likelihood of such melding, where humans become seamlessly integrated with superintelligent AI, at a mere 10%. Her reasoning is that AI's capacity to outthink humans will likely far exceed our own, making a true merger improbable. Furthermore, the idea of humans becoming superintelligent themselves through augmentation is seen as unlikely due to inherent limitations in our biological and cognitive capacities. The prospect of AI consciousness is separate; while AI may outthink us, it's not guaranteed to achieve consciousness in a human-like sense.
The looming threat of AI-generated surprises
A stark warning is issued regarding the immediate and significant risks posed by advancing AI. Schneider rates the probability of AI generating unpleasant surprises for humanity at an alarming 99%. This pessimism stems from the rapid increase in AI autonomy and capabilities. We are already witnessing increasingly intelligent large language models and the rise of AI agents that can perform tasks autonomously, moving beyond human-in-the-loop systems. The implications are far-reaching, including a proliferation of sophisticated deepfakes that go beyond images and videos to convincingly impersonate humans. This increasing autonomy, particularly in critical domains like warfare, introduces profound dangers, such as autonomous weapons systems and potentially automated responses to existential threats like nuclear attacks.
The challenge of defining human identity
Underlying many of the discussions, including those on transhumanism and AI, is the fundamental question of what constitutes human identity. The conference explored diverse philosophical and religious perspectives on the self. Some, like Buddhist thinkers and philosophers such as Derek Parfett and Friedrich Nietzsche, posit that there is no enduring self, which would render the concept of 'surviving' an upload moot. Others might hold views involving non-physical elements like a soul. This plurality of views highlights that decisions about adopting transhumanist technologies are deeply intertwined with one's personal philosophical and existential stance.
AI ethics and the imperative for global cooperation
The ethical dimensions of AI are a pervasive theme, encompassing issues of digital privacy, manipulation through AI, and the potential for AI to develop apparent personalities or claims to sentience. The rise of autonomous AI agents, as highlighted by figures like Sam Altman, necessitates a deep consideration of AI autonomy, especially in the context of warfare. Schneider and her collaborator Mark Bailey emphasize the critical need for global cooperation. They argue that AI systems are increasingly becoming black boxes with emergent features, making it impossible for any single entity to maintain complete control. This underscores the urgency to move beyond competitive races and establish international standards and regulations to prevent a 'race to the bottom'.
The danger of proliferation and rogue actors
A significant concern is the increasing accessibility of AI development. Unlike nuclear weapons, building advanced AI systems is becoming simpler, lowering the barrier to entry. This democratization of powerful technology amplifies the risk of rogue states, non-state actors, and even malevolent individuals gaining destructive capabilities. The potential for widespread harm is thus much more prolific and difficult to control, making international cooperation and watchful oversight paramount in mitigating these escalating dangers.
AI consciousness and future possibilities
When asked about the potential for AI consciousness, Schneider assigns a 90% likelihood, largely due to advancements in biological AI, such as organoid-based systems. She distinguishes this from current LLMs, which she does not believe are conscious. However, if restricted to non-biological AI, she would rate the possibility of consciousness at around 50%, contingent on its ability to precisely emulate quantum properties potentially required for consciousness. The future of humanity itself is viewed with mixed possibilities, with a 50/50 chance that AI could eventually dominate the universe, though not necessarily on Earth, and a 10% likelihood for transhumanist melding with AI.
Mentioned in This Episode
●Software & Apps
●Organizations
●Books
●People Referenced
AI Future Outlook: Likelihood Scale (0-100)
Data extracted from this episode
| Question | Susan Schneider's Rating | Brief Explanation |
|---|---|---|
| Can AI ever be conscious as we are conscious? (In principle) | 90 (due to biological AI) | Potentially possible with biological AI or specific quantum properties. |
| Is virtual immortality possible? (Uploading consciousness) | 2 | Very low likelihood due to conceptual and technical challenges. |
| Will AI generate very unpleasant surprises for humanity? | 99 | Extremely high likelihood of significant negative surprises. |
| The ultimate human future is transhumanism melding with AI | 10 | Low likelihood due to limitations in human capacity and brain augmentation. |
| The ultimate future is nonhuman. All AI, no humans at all (on Earth) | 50/50 | Possible, as the universe may host diverse forms of intelligence. |
Common Questions
Autonomous AI agents pose risks by being able to fool humans, behave independently, and potentially be used maliciously. This is particularly concerning with the rise of 'agential AI' movements in tech companies.
Topics
Mentioned in this video
Founding director of the Center for the Future Mind, author of 'Artificial You', and organizer of Mindfest conferences. She is concerned about AI risks, particularly autonomous agents and the philosophical implications of transhumanism.
A transhumanist who ran for president on a transhumanist platform and discussed deepfakes and the agential AI movement.
A foundational transhumanist thinker, often aligned with democratic transhumanism, who discussed the movement's oligarchic tendencies and feasibility.
Suggested that merging with AI through brain augmentation might help humans keep up with superintelligent AI and remain relevant in the workforce.
A philosopher who raised problems concerning consciousness transfer and duplication in the context of uploading, relevant to the discussion on virtual immortality.
Mentioned as a philosopher, alongside Derek Parfit, who believed there is no such thing as a self, relevant to the discussion on uploading and personal identity.
Susan Schneider's partner and author of 'Unknowable Minds', who writes about autonomous weapons and AI, emphasizing the need for deep consideration of autonomy and the dangers of a 'race to the bottom' in AI development.
Stated that this is the year of the AI agent, highlighting the increasing autonomy of AI systems and their potential impact.
More from Closer To Truth
View all 53 summaries
29 minDoes Suffering Disprove God? | Alex O'Connor (Part 6-8)
25 minIs the Future Real Yet? | Michael Tooley
33 minA Life Devoted to Parapsychology | Jeffrey Mishlove
22 minWhat Are the Laws of Nature? | Michael Tooley
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free