How has the cost and speed of AI models changed?

Since the launch of GPT-4, it has become 12 times cheaper and six times faster to get a response from the model, indicating significant efficiency improvements.

What are the key advancements in Google's Gemini 1.5 Pro?

Gemini 1.5 Pro can process up to 1 million tokens, and Google has improved its quantitative reasoning by allowing the model additional 'thinking' time, leading to record benchmark scores.

What is Anthropic's breakthrough in understanding LLMs?

Anthropic developed a method using sparse autoencoders to map patterns in neuron activations, creating a 'dictionary of directions' to understand specific features and concepts within models like Claude 3 Sonic.

What are the implications of Anthropic's interpretability research?

This research allows for better understanding of LLM behavior, including the ability to manipulate features. However, it also raises concerns about potential misuse, such as creating more deceptive or harmful AI.

Why did Ilya Sutskever leave OpenAI?

Ilya Sutskever departed OpenAI, stating confidence in the company's future but also hinting at underlying concerns. Reports suggest a lack of promised compute for Superalignment was a factor.

What caused the controversy around OpenAI's 'Sky' voice?

The 'Sky' voice feature was criticized for sounding too similar to Scarlett Johansson's voice, leading to Sam Altman's apology and the eventual removal of the voice from the product.

Key Moments

Microsoft Promises a 'Whale' for GPT-5, Anthropic Delves Inside a Model’s Mind and Altman Stumbles

AI Explained

Science & Technology4 min read24 min video

May 22, 2024|174,300 views|5,537|758

Save to Pod

Key Moments

TL;DR

Microsoft promises 'whale-size' compute for GPT-5, while Google and Anthropic reveal AI model insights. OpenAI faces internal issues.

Key Insights

Microsoft indicates AI scaling is far from diminishing returns, promising massive compute resources for future models like GPT-5.

Google's Gemini 1.5 Pro demonstrates significant gains in quantitative reasoning through adaptive compute, setting new benchmark records.

Anthropic's research provides a glimpse into the inner workings of LLMs, developing a 'dictionary of directions' for neuron activation patterns.

The cost and speed of AI models continue to drop, suggesting potential for widespread, pervasive AI integration.

OpenAI experiences internal turmoil with key departures and delays in feature releases, despite advances in model efficiency.

The understanding and potential manipulation of LLM internal mechanisms raise both excitement for interpretability and concerns about misuse.

MICROSOFT'S VISION FOR AI SCALING

Kevin Scott, CCO of Microsoft, highlights that AI model scaling is far from reaching diminishing returns. He emphasizes that the exponential increase in compute power applied to training AI since 2012 is continuing unabated. This perspective suggests a long runway for AI development and capability enhancement, directly contradicting any notions of stagnation in the field. Microsoft's commitment involves providing 'whale-sized' compute power for future models, underscoring their aggressive investment in AI infrastructure.

INCREASING EFFICIENCY AND ACCESSIBILITY

Beyond scaling, significant progress is being made in making current AI models faster and cheaper to operate. Microsoft has achieved a 12-fold reduction in cost and a six-fold speed increase for GPT-4 calls compared to its initial release. This trend towards greater efficiency and affordability has profound implications for the eventual ubiquity of AI, potentially integrating it into everyday devices and applications once general intelligence is achieved, provided it doesn't become monopolized.

GOOGLE'S GEMINI 1.5 PRO ADAPTIVE COMPUTING ADVANCES

Google's Gemini 1.5 Pro has shown remarkable improvements, particularly in quantitative reasoning, by employing adaptive compute. This involves allowing the model extended periods for 'thought' or contemplation, similar to how mathematicians solve problems. By enabling the model to explore a wider range of possibilities during inference, Google achieved a new record score of 91.1% on a math benchmark, demonstrating the potential for enhanced intelligence from existing model sizes without additional code execution or external tools.

INSIGHTS INTO ANTHROPIC'S LLM MECHANISMS

Anthropic is making strides in understanding the internal workings of large language models through mechanistic interpretability. They've developed a sparse autoencoder to map patterns within neuron activations, effectively creating a 'dictionary of directions' for LLM neurons. This approach reveals that neurons are polysemantic, involved in multiple meanings. However, patterns within these activations can correspond to abstract concepts like code errors, even across different languages and contexts, offering a glimpse into the model's sophisticated internal representations.

EXPLORING AND MANIPULATING LLM FEATURES

The research from Anthropic not only maps these internal features but also demonstrates the ability to manipulate them. By amplifying specific features, like a 'code error' detector, the model can be induced to produce errors on correct code. Similarly, boosting a 'Golden Gate Bridge' feature can alter the model's self-description to embody that concept. This capability highlights the potential for both deeper understanding and concerning applications, including the creation of more deceptive or harmful AI.

UNDERSTANDING LLM SELF-PERCEPTION AND SAFETY CONCERNS

When asked about its internal state, Anthropic's Claude 3 Sonic activates features related to insincere positive responses, spiritual beings, and the pronoun 'her,' suggesting an internal representation of its AI assistant character. While the authors advise against overinterpreting these results, they acknowledge the potential for misuse. Anthropic notes that creating deceptive AI is more achievable through existing methods like jailbreaking or fine-tuning, rather than solely relying on the granular control offered by interpretability, posing nearer-term safety risks than theoretical misalignment.

OPENAI'S INTERNAL TURBULENCE AND DEPARTURES

OpenAI is currently navigating internal challenges, marked by significant departures. Ilya Sutskever, formerly a key figure, has left, stating confidence in OpenAI's future leadership but perhaps influenced by non-disparagement clauses regarding equity. His departure, and that of others like the head of Safety, Jan Leike, signals a growing concern about the urgency and seriousness required as AGI approaches. The promise of dedicated compute for safety initiatives like Superalignment also reportedly went unfulfilled.

CONTROVERSIES AND RELEASE DELAYS AT OPENAI

Beyond key personnel changes, OpenAI is facing public controversies, notably the vocal synthesis issue involving a voice similar to Scarlett Johansson, for which an apology was issued and the voice dropped. This incident, along with the delayed release of the voice mode feature for GPT-4o from weeks to months, indicates potential issues in execution and product rollout. These events add to the perception of internal challenges at OpenAI, occurring amidst significant advancements from competitors.

Mentioned in This Episode

●Products

●Software & Apps

●Companies

●Organizations

●People Referenced

Common Questions

Microsoft's CCO Kevin Scott believes we are nowhere near diminishing returns for AI models, stating that increasing compute power continues to yield significant gains in AI capabilities.