Production AI Engineering starts with Evals

The speaker's current company, an end-to-end developer platform for building AI products, centered on an evaluation-driven workflow. It evolved from an eval tool to a debugger and eventually an IDE-like playground.

A cybersecurity company where Alena's father was president before joining Cloudflare, managing billions in revenue.

The speaker's first internship, working on Bing's distributed compute infrastructure. The experience was impactful but lacked intense creativity and room for interesting work.

Amazon's cloud data warehouse, noted for having a feature similar to Snowflake's variant type called 'Super'.

A web infrastructure and website security company where Alena's father is currently the president.

A platform for building, training, and deploying machine learning models. The speaker became a top non-employee contributor, working on document QA models.

A leading HTAP (Hybrid Transactional/Analytical Processing) database. The speaker was its first VP of Engineering and discussed its advanced technology but also its high cost and niche market suitability, comparing its evolution to Neon for wider adoption.

An online personal styling service, mentioned as a company Impira tried to close a deal with early on.

The speaker's first AI company, founded on the idea of making unstructured data as easy to use as structured data, leveraging ML models. They learned critical business lessons about sales, customer empathy, and market fit.

Mentioned in the context of Figma's acquisition by Adobe, a factor contributing to Figma's sense of stability.

One of the early movers in the AI tooling space, offering durable playgrounds, prompt saving, and eval features, but predating Brain Trust's focus on engineering efficiency and declarative evals.

A self-driving car company, where Eden, Brain Trust's Head of Product, was a designer.

A design tool company that acquired Impira. The speaker was there for eight months and discussed the challenges of integrating AI, especially visual AI, into a high-quality product with an annual release cycle.

A company founded by Nikita Shamgunov, aiming to provide a hyper-inexpensive PostgreSQL offering with a world's best free tier, contrasting with SingleStore's high-cost model.

A customer of Brain Trust, whose engineers expressed a desire for Brain Trust's playground to become their IDE.

A data and AI company with a famous hybrid on-prem model. While their model is successful, it's often viewed with mixed perspectives, serving as a cautionary tale/inspiration for Brain Trust's hybrid approach.

A monitoring and analytics platform that Figma used, mentioned as a comparison point for observability solutions.

Mentioned as a public cloud provider alongside Amazon and Azure, suggesting big companies have special relationships for their AI models.

A cloud-based data warehousing company, praised for its best-in-class implementation of semi-structured data with its 'variant type' but criticized for its expensive packaging and high minimum query time.

A leading AI research and deployment company. Brain Trust leveraged their API for LLM judging and tool interactions, and OpenAI's models are heavily adopted by Brain Trust customers due to reliability and availability.

A platform for developing mobile and web applications. The interviewee's analogy for Brain Trust is similar to Firebase for traditional software developers, providing an end-to-end platform.

A vector database company, mentioned as a company that the speaker would host on their podcast to hear an opposing view on vector databases.

The company behind LLaMA models, seen as a key player invigorated by OpenAI's advancements and contributing to a healthier AI ecosystem.

An AI product that records screen activity and titles it properly, delighting users with a small AI sprinkle of intelligence.

A tool that allows users to run browsers in the cloud, integrated into Brain Trust for defining custom tools.

An automation platform and early adopter of Brain Trust, whose engineer Brian provided critical feedback during product development. They also use linear for auto-generating ticket titles.

An AI safety and research company, known for its Claude models. The speaker considered starting a company based on their eval process while interviewing there. Praised for offering strong alternatives to OpenAI, especially with Haiku.

A workflow as code platform, where a cloud VPC peering solution was used, demonstrating complex deployment scenarios addressed by Brain Trust's hybrid model.

A cloud platform for serverless deployment, a customer and investor in Brain Trust. Malta, an engineer from Vercel, provided a key quote on workflow transformation.

A data platform for security, observability, and IT operations, mentioned as a company that built its own database technology to solve the "variant type" problem for observability.

Microsoft's search engine, where the speaker worked on distributed compute infrastructure that is now part of Azure.

A column-oriented database management system for online analytical processing, noted for working on something similar to semi-structured data handling.

A powerful, open-source object-relational database system that Neon is building a hyper-inexpensive version of.

A groundbreaking convolutional neural network, whose release made the speaker realize new possibilities for data processing, inspiring Impira.

An advanced language model by OpenAI, which 'totally blew the speaker's mind' with its ability to extract information from unstructured text, even without visual signals, surpassing LayoutLM.

An early AI adopter that iteratively ships AI features. Its conference was mentioned, and its AI features were initially hacked by founders at a retreat.

A popular code editor from Microsoft. The speaker entertains the idea of forking it as a future direction for Brain Trust's IDE-like playground.

A spreadsheet application, used as an analogy for basic, DIY eval systems that many early AI products resemble. Brain Trust aims to offer more advanced features beyond simple spreadsheet functionality.

A model from the Claude 3 family, noted as a smart, cheap, and fast model with tool-calling capabilities, providing a significant foothold for Anthropic.

A collaborative online document editor, used as an analogy for the collaborative and real-time nature of Brain Trust's playground.

Anthropic's family of large language models. Claude 3, particularly Haiku and Sonnet, offered compelling alternatives to OpenAI for Brain Trust's customers.

A secure runtime for JavaScript and TypeScript, mentioned in the context of different ways of handling arbitrary code execution in JavaScript.

An open-source machine learning framework. PyTorch is seen as a huge improvement over TensorFlow, but still not ideal for TypeScript engineers.

An open-source, in-memory data store, mentioned as an example of a database technology that solves specific technical problems.

Microsoft's cloud platform. Big companies using OpenAI models often use Azure due to special relationships, but it can present engineering challenges compared to OpenAI's direct endpoints.

An observability platform that built its own super wide column store. The speaker agrees with their decision given the lack of accessible semi-structured data solutions like Snowflake's variant type.

A neural network-based technique for natural language processing pre-training, which significantly accelerated text-based information extraction and began to cannibalize Impira's computer vision-based approach.

A company that provides a search API, integrated into Brain Trust as a custom tool for agents to search the internet.

A document collaboration platform and early Brain Trust user. They started using the product quickly due to its hybrid on-prem model and focus on evaluations.

Anthropic's latest family of models. Its release, particularly Haiku and Sonnet, significantly shifted market share, allowing developers more options beyond OpenAI.

OpenAI's multimodal flagship model, whose capabilities are leading to a shift where complex reasoning and agentic logic will be integrated directly into the model, rather than requiring external frameworks.

An AI chatbot. The speaker mentioned Dylan Patel's calculations on whether Grok is burning cash.

An in-memory, column-oriented, relational database management system, mentioned as a more expensive alternative to SingleStore in its early days.

An open-source embedding database, mentioned as a company that the speaker would host on their podcast to hear an opposing view on vector databases.

A framework for programmatically optimizing prompts, mentioned as a step towards automatic optimization, but criticized for its PyTorch-like code structure that might not suit TypeScript engineers.

A modern, fast web framework for building APIs with Python, mentioned as a common tool for building applications that can benefit from sprinkled intelligence.

An in-process SQL OLAP database management system, mentioned for its struct type, which has downsides compared to Snowflake's variant type for handling schema changes.

An AI chatbot developed by OpenAI, whose release validated the speaker's pivot away from Impira's vision-based document processing, as it could cannibalize their technology.

A model from the Claude 3 family, initially seen as a 'middle child' but now considered both cheap and smart, offering a pleasant communication experience.

A Python library fundamental for numerical computing, mentioned as intuitive for continuous mathematicians (ML people) but non-intuitive for the speaker (software engineer) who prefers discrete math.

An AI-powered code editor, positioned as complementary to Brain Trust: Cursor enhances traditional software engineering with AI, while Brain Trust brings software engineering best practices to AI development.

A JavaScript runtime, which Brain Trust settled on supporting for arbitrary code execution, similar to Vercel's approach.

A React framework for web development, used as an example of a common tool for TypeScript engineers.

A project management tool used at Brain Trust, which integrates with AI to auto-generate ticket titles from Slack complaints, illustrating a single-prompt manipulation use case.

OpenAI's interactive tool for experimenting with language models. Brain Trust's playground was built to be more durable, shareable, and collaborative than this.

A database-spreadsheet hybrid software and early Brain Trust user, whose data stayed in their cloud and benefited from the hybrid on-prem model.

An extension for PostgreSQL that adds support for geographic objects. The speaker's ideal database scenario would be a unique combination of storage and execution paradigms, like PostGIS.

An open-source machine learning framework, whose coding style for automatic optimization is seen as non-ergonomic for TypeScript engineers.

A cloud platform. Customers often have committed spend with AWS, which would ordinarily suggest using cloud services for models, but the speaker notes that direct OpenAI endpoints are often preferred for convenience and access to newest models.

Meta's family of open-source large language models. LLaMA 3 8B is mentioned as a powerful open-source model that could change the fine-tuning landscape, with many people incentivized for its success.

Original founder of SingleStore, who is now trying to make database technology more accessible and inexpensive with Neon, in contrast to SingleStore's high-end market.

Co-founder of Notion, who hacked the first versions of Notion AI at a retreat.

Co-founder of Notion, who hacked the first versions of Notion AI at a retreat.

An analyst who publishes calculations on the profitability of AI companies, mentioned in the context of GPU inference market and Grok's cash burn.

Mentioned as a fellow podcaster who would cynically say that evals are not new to ML engineering, but rather a standard practice.

An investor in Impira and Brain Trust, who provided significant help during the Impira acquisition process and insight into the rise of AI engineering, supporting the speaker's decisions.

An engineer from Vercel who used to work on Google search. He provided a very positive quote about Brain Trust's ability to transform engineering workflows and discovered many bugs while preparing a talk on Brain Trust.

The CEO of Vercel, referred to as 'GMO', who gave important advice on making technology bets as a startup and solving arbitrary code execution.

A high-performance database machine, mentioned as a more expensive alternative to SingleStore in its early days.

A feature in Figma that represents a convergence between UI engineering and design, potentially empowering designers with AI for code generation.

A multimodal pre-trained model for document image understanding, which Impira retooled its product to use after seeing its effectiveness, although it was later surpassed by GPT-3.

Continuous Integration/Continuous Deployment, used as an analogy by a VC who initially rejected Brain Trust, viewing evals as a 'crappy market' akin to CI/CD, before becoming interested in their observability features.

A category of tools for managing large language models in production. The speaker preferred to focus on evals first due to the significant pain points, rather than going directly into logging and observability.

An agent that writes, persists, and reuses code for future tasks. The speaker notes that while code generation is being used, no one seems to be doing the full Voyager-style code persistence and reuse in production.