Building AGI in Real Time (OpenAI Dev Day 2024)

A process of customizing a pre-trained AI model on specific data to tailor it for a particular task or company's communication style and brand guidelines, now extended to include vision.

A technique to create a smaller, more efficient version of a large, powerful AI model while retaining similar capabilities, making AI more accessible and runnable on less powerful hardware.

A UI framework for Apple platforms that, alongside Swift, has simplified iOS development, allowing AI models to assist in architecting and generating code.

A capability that allows AI models to access and utilize external tools and information, enabling them to pull up details from databases or interact with other services in real-time demos.

A communication protocol used by the Real-time API to establish persistent, bidirectional connections, allowing AI to respond instantly and handle interruptions in conversations.

Applying fine-tuning techniques to image-based AI models, with applications in medicine for diagnosis and improving accuracy in tasks like object detection (bounding boxes) on bespoke formats.

A term Sam Altman believes has become a buzzword, suggesting a shift in focus to continuous AI improvement rather than defining a specific AGI threshold.

Head of Product for the OpenAI platform, who led the Dev Day keynote, introduced new features, and discussed the global outreach strategy for developers and design choices behind new APIs.

The CEO of OpenAI, who discussed the evolving understanding of AGI, emphasized responsible AI development, and participated in a Q&A session.

An OpenAI Developer Experience Engineer and 'strawberry shop owner' in a demo, who showcased the Real-time API's voice mode and function calling by role-playing a strawberry vendor for a phone order.

A member of the OpenAI API team who discussed structured outputs and the voice mode API, sharing insights into the design decisions for the new websocket API.

OpenAI's Chief Product Officer, who joined Sam Altman for a discussion on the future of AI and the big picture for the company.

OpenAI's Head of Developer Experience and AI Engineer, who showcased live demos of the International Space Station tracker and a drone, and offered advice on consuming coding models and real-time APIs.

A super blogger and returning guest co-host who live-blogged Dev Day, praised the real-time API's potential for web apps, and shared his own AI experiments and feature requests.

Co-founder of Co-op Labs, AI engineer, and World's Fair closing keynote speaker, who talked about fine-tuning GPT-4o to achieve high scores on S-bench and his techniques for generating human-like reasoning traces.

Used as an analogy to describe OpenAI's expanding role beyond a model provider to an 'AI Cloud,' offering comprehensive services like storage and compute.

A smaller version of GPT-01, noted for its excellent performance in math, coding, and STEM subjects, making it suitable for specific, rooted-in-code tasks.

A code editor that Michelle Pocas still uses with Copilot, despite trying out Cursor, highlighting it as her tool of choice.

An AI model used for transcribing audio, specifically mentioned in the context of processing hour-long YouTube videos before multimodal capabilities were available.

Mentioned as a point of reference for the exponential growth in AI capabilities, suggesting that GPT-01 represents a similar 'scale moment' in AI development.

A software engineer agent from Cognition mentioned as utilizing 01 Preview in its own software, similar to how Cursor integrates OpenAI models.

A travel app originally developed by Simon and Gis, later modified by Roman Huge to incorporate voice components and real-time calling abilities for the Dev Day demo.

A text editor that was controlled with voice mode in an internal hackathon project, demonstrating new ways of interacting with code.

A company mentioned as offering 'code interpreter as a service,' providing sandboxed environments for running and compiling code.

A previous OpenAI language model, compared to GPT-01 which surpasses it in advanced math and complex coding, but GPT-4 is still suitable for tasks like screenplay writing.

A coding tool mentioned as a preferred environment for interacting with OpenAI's coding models, particularly GPT-01 Preview and Mini.

An AI coding assistant that was the primary reason Michelle Pocas joined OpenAI, indicating its significant impact on her work.

An API mentioned as a specific use case for the Assistants API, praised for its ability to run and compile code within a sandboxed environment.

Google's notebook product, admired by Sam Altman for its cool format and ability to generate podcast-style voices, allowing users to create dynamic content from their documents.

An application doing 'cool things' with language translation, showcasing the practical applications of AI models in real-world scenarios.

An online forum where a user highlighted a detail about the Real-time API providing a text version of spoken content for storage and analysis.

An OpenAI feature that offers discounts when the AI sees a prompt it has processed before, making AI usage more affordable and efficient without requiring code changes from developers.

The API used in Elon Biegio's demo to make phone calls with AI agents, integrating voice mode and function calling.

One of two applications showcased by Roman Huge on stage, demonstrating the capabilities of OpenAI's models in real-time.

A programming language that, along with Swift UI, has made iOS development easier, especially when combined with AI coding partners like GPT-01.

A partner solution mentioned for helping developers integrate with the real-time API, offering native plugging capabilities for various client-side and server-side voice interactions.

A competitor's AI model praised for its bounding box capabilities and its ability to process long video inputs by slicing them into individual frames.

Co-op Labs' model, capable of software engineering tasks, which was developed using specific fine-tuning techniques and data pipelines to generate human-like reasoning traces, outperforming GPT-01 out of the box on S-bench.

A JavaScript library mentioned in the context of Genie's performance in UI development, where evaluating the model's output could be challenging without Vision fine-tuning.

A new API announced by OpenAI designed for real-time interaction with AI, using persistent WebSocket connections and function calling to enable more natural, interruptible voice conversations.

OpenAI's new model, described as a significant leap forward, trained with reinforcement learning to reason and learn from mistakes, excelling in advanced math and complex coding problems.

Mentioned as an example of an existing voice mode, with potential future extensions for real-time video and image capabilities.

An IDE mentioned as a development environment for iOS apps, where GPT-01 and ChatGPT are used as coding and brainstorming partners due to its less deep integration.

A client-side technology suggested for developers who want very robust real-time communication, potentially as an alternative to directly working with WebSockets at scale.

A platform described as the 'Metamask for AI,' allowing users to bring their own API keys and securely manage their AI usage across different models.

A model that Co-op Labs successfully fine-tuned to achieve higher scores than GPT-01 on S-bench, demonstrating that older models can be enhanced through custom reasoning.

Mentioned as an example of how people globally prefer voice notes for communication, highlighting the natural human preference for spoken interaction over typing.

A company doing 'cool things' with coding, demonstrating advancements in AI applications for legal tech.

Platform where Sam Altman saw examples of NotebookLM and its functionalities, and where his philosophy of iterative deployment was cultivated during his time there.

A rival AI company mentioned regarding its prompt caching duration (5 minutes) and 'projects' feature, which is admired by Kevin Will.

Ali Pullin's company, distinguished for its work in fine-tuning GPT-4o with synthetic datasets and for obscuring Chain of Thought traces as intellectual property.

An autonomous driving company, whose self-driving car experience was used as an analogy to describe how quickly humans adapt to new AI capabilities, initially amazed then quickly accustomed.

The second application demonstrated by Roman Huge, highlighting the coding models' ability to architect and wire up specific functionalities like frontend, backend, UDP, and WebSockets.

A partner solution like LiveKit, offering ways for developers to get started with OpenAI's real-time API, depending on specific use cases.

One of the US states that has partnered with OpenAI, demonstrating initial government adoption of AI tools.

Another US state mentioned as partnering with OpenAI, reflecting the growing collaboration between AI companies and government agencies.

One of the international locations, along with San Francisco and London, where OpenAI planned to hold events to meet more developers, indicating a global outreach strategy.

An organization mentioned as partnering with OpenAI, indicating efforts to leverage AI for global development and humanitarian challenges.

Mentioned as a group like Case Text, doing 'cool things' with coding, suggesting its involvement in advanced AI applications.

A benchmark for AI models, where Co-op Labs' Genie model achieved high scores by fine-tuning GPT-4o, though its reasoning traces were withheld, mirroring OpenAI's later competitive approach.