Why is a 'fault-first mindset' important for AI engineers?

Language models have unpredictable latency and response formats, unlike conventional APIs. A fault-first mindset helps engineers build robust systems that can handle these uncertainties, ensuring a reliable user experience.

How does building with AI models differ from traditional software engineering?

AI engineering often involves working with unpredictable mediums like language models, which contrasts with the predictability of traditional APIs and databases. It requires skills from distributed systems engineering, like careful error handling and concurrency, applied at the application level.

What technical approaches does Elicit use for AI engineering?

Elicit utilizes standard techniques such as retries, fallbacks, timeouts, and careful error handling. They also emphasize strong typing across their tech stack (Python to TypeScript) and use checked exceptions in Python to manage potential errors.

How can companies manage AI development in larger organizations?

For larger organizations, especially those with compliance needs like HIPAA, creating an internal AI gateway or platform to standardize model usage and manage security can be feasible and sensible.

Is prompt engineering a durable skill for AI engineers?

The speaker suggests that while prompt engineering might evolve, the durable skill will be in how the overall ML problem is structured and the right questions are asked, rather than the specific phrasing of prompts.

How should companies assess new AI model capabilities?

Companies should evaluate new models based on their performance on specific tasks, using comprehensive evaluation frameworks that measure accuracy, confidence, performance, and cost, and consider their potential for new features or applications.

What is the 'ML-first' mindset in software development?

An ML-first mindset involves relinquishing some control, embracing the opaqueness of models, and leveraging their inherent uncertainty to create powerful, unexpected outcomes, rather than over-constraining problems with traditional software patterns.

Where can I find resources to learn about AI and ML?

Elicit provides an ML reading list, categorized into tiers, which serves as a valuable resource for those seeking to build knowledge in AI/ML. Some communities also host paper clubs for asynchronous discussion.

How can I source AI engineers effectively?

Effective sourcing involves both outbound efforts (reaching out to experienced engineers on LinkedIn, identifying side projects) and inbound strategies (raising a beacon through content like blog posts, open-source projects, and clear job descriptions) that attract talent interested in the company's mission.

What are good places to find junior AI engineering talent?

For less experienced individuals, casting a wide net through platforms like hackathons, challenges, and engaging in communities (Discord, Slack) can help identify and attract enthusiastic candidates with strong conventional engineering skills and a passion for AI.

How should AI engineer interviews be structured?

Interviews should simulate the actual work, focusing on technical capability, a fault-first mindset, and cultural fit. It's crucial to offer candidates a realistic view of the role and environment, allowing them to assess the company as much as being assessed.

Key Moments

How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit)

Latent Space Podcast

Science & Technology4 min read69 min video

Jun 21, 2024|1,191 views|33|4

Save to Pod

Key Moments

TL;DR

Hiring AI engineers requires a blend of software skills, ML curiosity, and a fault-tolerant mindset, with interviews simulating real work.

Key Insights

AI engineering is approximately 90% software engineering with a critical 10% specialized AI/ML knowledge.

Key attributes for AI engineers include strong conventional software engineering skills, curiosity for ML/LLMs, and a "fault-first" mindset for building resilient systems.

LLMs introduce unpredictability in latency and response content, necessitating robust error handling, retries, fallbacks, and strong typing.

Interview processes should simulate real-world tasks, focusing on defensive programming and system design that accounts for potential failures.

Curiosity for LLM capabilities is crucial, but it should be tied to product goals and user needs, not just technological advancements.

Sourcing AI engineers involves a mix of clear employer branding, outbound outreach, and engagement in relevant communities like hackathons and specialized job boards.

DEFINING THE AI ENGINEER ROLE

The role of an AI engineer blends conventional software engineering with specialized AI/ML knowledge. James Brady, Head of Engineering at Elicit, notes that while AI engineering is often described as 90% software engineering, the remaining 10% is highly differentiated and critical. This blend stems from the need to build robust applications on top of inherently unpredictable technologies like large language models (LLMs).

CORE SKILLS: SOFTWARE ENGINEERING AND FAULT TOLERANCE

Effective AI engineers possess strong conventional software engineering skills as a baseline. Crucially, they must also have a "fault-first" mindset, meaning they proactively build systems that can handle failures. This is essential due to the high variability in LLM latency and response content, requiring techniques like retries, fallbacks, and careful error handling.

NAVIGATING LLM UNPREDICTABILITY

Working with LLMs introduces significant challenges compared to traditional APIs. Latency can vary by a factor of ten, and response formats and semantics are naturally unpredictable. AI engineers need to build resilient applications that provide a stable user experience despite this underlying chaos. This often involves applying principles from distributed systems engineering at the application level, such as strong typing and checked exceptions, to manage data shapes and potential errors.

INTERVIEWING FOR THE RIGHT MINDSET

Interviewing AI engineers requires moving beyond traditional happy-path coding challenges. Technical exercises should incorporate adding features and fixing bugs within a codebase that simulates the unpredictable nature of LLMs. System design interviews can effectively probe for a fault-first mindset by posing hypothetical failure scenarios, such as node failures or network slowness.

CULTIVATING CURIOSITY AND PRODUCT FOCUS

A genuine curiosity and enthusiasm for machine learning and LLM capabilities are vital. However, this must be coupled with a product mindset. AI engineers should be excited by new models and features but always frame their exploration around how these advancements can solve user problems or improve the product's strategic goals, rather than just pursuing them for their own sake.

SOURCING TALENT AND EMPLOYER BRANDING

Effective sourcing for AI engineers involves a multifaceted approach. This includes maintaining an active online presence through blogs and social media, engaging in relevant communities (hackathons, conferences), and conducting targeted outbound outreach. For smaller organizations, demonstrating a clear mission, great teammates, and a compelling product through employer branding is crucial for attracting top talent.

THE ML-FIRST MINDSET ADJUSTMENT

Adopting an "ML-first" approach requires a significant mindset shift, moving away from the need for complete control over every component. It involves relinquishing some control to opaque black-box models and developing comfort with unexpected outputs. While this can lead to powerful emergent capabilities, it necessitates careful integration, often using regular expressions or other post-processing for validation, balancing innovation with necessary robustness.

BALANCING INNOVATION AND STANDARDIZATION

In the rapidly evolving AI landscape, there's a natural tension between the need for rapid experimentation and the desire for standardization. While large organizations might benefit from AI gateways for control and security, smaller, agile teams often need the flexibility to quickly switch between models and prompts. Finding the right time to introduce abstractions and standards is a key judgment call, especially in this "wild west" phase of AI development.

THE EVOLVING ROLE OF PROMPT ENGINEERING

While prompt engineering is currently a key skill, it's unlikely to remain a durable differentiator. Instead, the ability to structure ML problems and ask the right questions will likely become more crucial. The operational challenges of LLMs, such as managing latency and handling unpredictable inputs, will continue to demand defensive engineering Socratic methods of inquiry into their capabilities.

ASSESSING CANDIDATE MATURITY AND FIT

Modern young professionals often exhibit a higher degree of maturity, capability, and drive than previous generations. Identifying and nurturing this talent is key. The interview process should simulate collaborative work, allowing candidates to assess the company as much as the company assesses them, particularly important for an emerging field like AI engineering where established playbooks are still being written.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Concepts

●People Referenced

Common Questions

An effective AI engineer needs strong conventional software engineering skills, a genuine curiosity and enthusiasm for machine learning and language models, and a fault-first mindset to build resilient systems.

Topics

AI & Machine Learning Technology & Innovation Language Models AI Engineering Career Transition Software Development Machine Learning Career & Skills Hiring Process Fault Tolerance Sourcing Talent

Mentioned in this video

People

Sam Bankman-Fried

Mentioned in relation to the impact of his downfall on the Effective Altruism (EA) community, prompting reflection and caution.

James Brady

Head of engineering at Elicit, discussed his transition from traditional VP of Technology roles to AI due to the generational shift in AI/ML capabilities.

Adam Wiggins

Co-founder of Heroku, previously worked on Muse, and currently acts as an internal journalist for Elicit, focusing on supporting James's articles and learning about AI applications.

Software & Apps

Elicit

A company focused on applying language model capabilities to science and literature search.

Typescript

A programming language used at Elicit for front-end development, sharing types with backend Python code via OpenAPI specs.

Stripe API

Used as an example of a conventional API with predictable latency, contrasted with language model APIs.

Claude 3.5

A new model released by Anthropic, discussed regarding its capabilities and how to assess them.

React

A JavaScript library for building user interfaces, mentioned as an example of a technology that eventually led to standardized abstractions after a period of rapid evolution.

Heroku

A cloud platform as a service founded by Adam Wiggins, which standardized the early development experience.

LangChain

Mentioned in the context of discussions around standardization and abstractions in the rapidly evolving AI space.

Companies

Muse

A previous venture Adam Wiggins worked on.

OpenAI

Mentioned as a provider of language models; discussed in the context of API similarities and potential fallbacks.

Anthropic

A company that recently released Claude 3.5, discussed in the context of model capabilities and potential fallbacks.

Concepts

OpenAPI

A specification used at Elicit to generate TypeScript types dynamically from Python definitions, facilitating type sharing between backend and frontend.

Effective Altruism

A philosophy and social movement focused on using evidence and reason to do the most good. Its community is undergoing reflection after the issues related to Sam Bankman-Fried.

Organizations

80,000 Hours

An organization whose job list Elicit has used to hire individuals interested in AI safety and effective altruism.