Key Moments

Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space PodcastLatent Space Podcast
Science & Technology6 min read99 min video
Nov 16, 2026|2,149 views|13
Save to Pod
TL;DR

Black Forest Labs has developed 'visual intelligence' models that learn from pixels, not just text, with applications ranging from realistic image/video generation to robotics, aiming to capture intuitive human understanding. They've raised $450M to achieve this by creating versatile, multimodal foundational models like Flux.

Key Insights

1

Black Forest Labs (BFL) is shifting from image generation to 'visual intelligence', aiming to imbue models with intuitive, human-like understanding of the physical world learned from pixels and natural signals.

2

BFL has raised $450 million to date, with their latest revenue information from December indicating they've surpassed $100 million in revenue, demonstrating significant commercial traction.

3

The Flux family of models, starting with Flux 1 and culminating in Flux 3, is multimodal, learning from images, video, and audio to understand and generate content, and is also being applied to robotics ('Flux Action').

4

Flux Context, a key BFL development, allows for image editing and manipulation through text and image prompts, with the potential to replace many traditional creative workflows.

5

BFL's research on video generation, including 'Align Your Latents' and 'Stable Video Diffusion', emphasizes the critical role of data filtering and temporal consistency in achieving high-quality outputs.

6

The company's approach to robotics focuses on 'one-shot learning' and 'contextual learning', aiming for robots to perform tasks after a single demonstration, a significant leap towards 'superhuman visual intelligence'.

From image generation to a new paradigm of 'visual intelligence'

Black Forest Labs (BFL), co-founded by Andreas Blattmann and Dustin Podell, has evolved significantly from its origins in image generation, stemming from the foundational work on Latent Diffusion and Stable Diffusion. Initially focused on creating high-quality image and video generation models, BFL has pivoted towards a broader concept they term 'visual intelligence'. This new direction aims to harness the intuitive, non-verbal understanding of the physical world that humans possess—like understanding physics implicitly when catching a ball—and translate it into pixel-based models. Unlike language models that process discrete tokens, BFL's approach focuses on 'natural signals' such as pixels and audio, learning directly from raw sensory data to build a more holistic understanding of reality. This foundational shift is powered by their Flux family of multimodal models, which are designed to learn from and generate across various modalities including images, video, and audio, with applications extending into robotics.

The Flux models: Multimodal learning for diverse applications

The Flux model series represents BFL's core innovation in multimodal learning. Starting with Flux 1 and progressing to the latest Flux 3, these models are trained on vast, diverse datasets encompassing images, videos, and audio. This comprehensive training allows them to grasp complex relationships within the data and perform a wide range of tasks. For instance, Flux 3 is not just for generating realistic images and videos but also for enabling robots to understand and interact with their environment through 'Flux Action'. The core idea is to build a foundational model that can serve as a versatile "brain" for various AI applications, from creative media generation to complex physical interactions in robotics. This multimodal approach is key to unlocking a deeper, more grounded understanding of the world, moving beyond purely symbolic or textual representations.

Commercial success and strategic evolution

BFL has achieved substantial commercial success, having raised $450 million in funding to date. Their latest financial disclosures indicate they have surpassed $100 million in revenue, underscoring the market's strong demand for their technology. While initially established as an image generation lab, BFL's strategic pivot towards visual intelligence and robotics was a conscious decision, not a change in strategy but an expansion of scope. The company emphasizes that their success is built on a foundation of deep research and a clear vision for the future of AI, aiming to create general-purpose, highly capable models rather than niche solutions. This evolution reflects a commitment to pushing the boundaries of what AI can achieve, leveraging their expertise in diffusion models and multimodal learning to tackle increasingly complex challenges.

Flux Context: Revolutionizing image editing and creative workflows

Flux Context is a prime example of BFL's innovation in practical AI applications. This model enhances image editing capabilities by allowing users to manipulate images using both text prompts and example images. This dual-prompting approach, combined with the model's ability to understand spatial relationships and context within an image, offers a powerful new tool for creative professionals. It can handle tasks like adding logos to existing images, maintaining character consistency across edits, and performing complex manipulations that previously required specialized software or multiple manual steps. BFL sees Flux Context as a significant step towards democratizing advanced image editing, making powerful creative tools more accessible and efficient for a wider range of users.

The critical role of data and training in video generation

BFL's work in video generation, including Stable Video Diffusion and their own video models, highlights the paramount importance of data quality and filtering. Andreas Blattmann emphasizes that even the most sophisticated algorithms will produce poor results if trained on low-quality, blurry, or static data. They explore the balance between data quantity and quality, noting that while large datasets are beneficial, meticulous filtering and curation are essential for effective training. Their research has also delved into techniques for achieving temporal consistency in video, ensuring smooth and coherent motion across frames, which is a fundamental challenge in generative video.

Advancing robotics with visually intelligent agents

A significant focus for BFL is the application of their visual intelligence models to robotics. They envision robots that can learn tasks through observation, akin to how humans learn. This 'one-shot' or 'few-shot' learning capability, where a robot can perform a new task after seeing it demonstrated just once or a few times, represents a major leap towards more capable and adaptable robotic systems. Their 'Flux Action' model, for instance, predicts actions based on video input and state information, enabling robots to perform tasks in both simulated and real-world environments. This push towards 'superhuman visual intelligence' in robotics aims to enable robots to understand and interact with the world with a level of intuition and adaptability previously unseen.

Technical innovations: Flow matching and diffusion transformers

BFL's technical contributions include advancements in generative modeling paradigms. They have explored and utilized concepts like flow matching, which offers a more mathematically tractable and efficient alternative to traditional diffusion models for learning vector fields between probability distributions. This approach, along with the use of diffusion transformers, allows for more efficient training and faster sampling. Their work on SD3, for example, integrated these advancements, demonstrating improvements in areas like text rendering within images and overall model performance. These technical innovations are crucial for scaling up the capabilities of their multimodal models.

The future of AI: Towards general-purpose, interactive intelligence

Looking ahead, BFL is focused on building highly general-purpose AI systems that can understand and interact with the world in a comprehensive way. They see a future where AI systems can seamlessly integrate various modalities—vision, audio, language, and action—to achieve a level of intelligence that rivals human cognitive abilities. The pursuit of this goal involves continuous research into foundational models, efficient training techniques, and novel applications, with a strong emphasis on responsible development and deployment. Their vision extends to making AI tools more accessible and beneficial, aiming to enhance human productivity and creativity rather than replace it.

Common Questions

Black Forest Labs' core mission has evolved from image generation to building 'visual intelligence,' focusing on pixel-based models and learning methods for images, video, audio, and action prediction. Their goal is to develop intuitive, System One intelligence for real-world interaction, especially in robotics.

Topics

Mentioned in this video

Software & Apps
ComfyUI

A UI developed for Stable Diffusion. The host notes he went down the path of building a UI for Stable Diffusion, while Dustin pursued research.

Project Mercury

A type of masking-style discrete diffusion language model, mentioned alongside Diffusion Gemma, where it jumps from a mask to a token in discrete space.

Diffusion Gemma

A text generation model based on diffusion, mentioned as an example of diffusion models for text that people can experiment with, offering good benchmarks and fast operation.

Stable Diffusion

A generative AI model that built upon Latent Diffusion. Its launch was a pivotal moment for many in the AI field, including the founders of Black Forest Labs, and the host himself.

SDXL

A foundation model by Stability AI (and later BFL contributors) aimed at making models enjoyable to use, supporting multi-aspect training and being the first to incorporate many common improvements from the open-source community.

Control LoRA

A generative AI technique, created by Dustin Podell, used to control specific aspects of image generation, mentioned as part of his contributions to the field.

SD3

A generative AI model that was the culmination of Stability AI's work, focusing on understanding which features are learned at different points in the data-to-noise trajectory to optimize training.

Flux Ultra

A model by Black Forest Labs, mentioned as one of the projects Dustin Podell was the post-training lead for.

Flux

An early generative model from Black Forest Labs, which surprised many with its capabilities upon release, having been trained in just three months.

Flux Context

An image editing model developed by Black Forest Labs, specifically designed to replace common creative workflows and editors by allowing image editing through natural language prompts and visual references.

Adobe Photoshop

A leading image editing software, mentioned as a type of workflow/editor that Flux Context models could potentially replace for common editing tasks.

Midjourney

An AI generative program, with its V1 release prompting Dustin Podell to realize the influential direction of AI. Later noted for its well-known Discord loop for feedback and style book features.

Ideogram

A generative AI model, with its V1 mentioned as being closed-source at the time of SDXL's development.

DALL-E 3

A generative AI model for images by OpenAI, mentioned as being closed-source when SDXL and early versions were being developed.

DreamBooth

An early style of fine-tuning generative models, described as very inefficient, used by a viral company that allowed users to generate images from their own photos.

Flux Reaction

The latest robotics version of Flux models, focused on action prediction.

ChatGPT

A large language model by OpenAI, mentioned in the context of autoregressive models and how they handle agentic loops.

PaLM

An open-source language model from Google, used as a comparison point for the data efficiency of Black Forest Labs' models in fine-tuning for robotics, requiring eight times less data.

Companies
NVIDIA

A technology company, where Andreas Blattmann previously worked on video generation before Stable Diffusion, publishing the 'Align Your Latents' paper.

Dyna Robotics

A robotics company mentioned as having a platform similar to what Black Forest Labs is developing, particularly in the context of robot evaluation.

Stability AI

An AI company that further developed Latent Diffusion, leading to Stable Diffusion. Key team members later spun off to form Black Forest Labs.

Black Forest Labs

A company formed by core image generation personnel from Stability AI, focused on creating world-class open and closed weights models for image and video generation, and now robotics. Named after the Black Forest region in Germany where it was founded.

Canva

A graphic design platform, mentioned as a type of workflow/editor that Flux Context models could potentially replace for everyday editing tasks.

OpenAI

A leading AI research and deployment company, mentioned as a previous employer of Lucas Beyer and where Team Dohrn also works.

Runway

An AI company producing world-class models for image and video generation, mentioned in relation to their pipeline-based second-generation models and also having a robotics platform.

Google

A major technology company, mentioned as a previous employer of Lucas Beyer.

Meta

A technology conglomerate, mentioned as the current employer of Lucas Beyer and the developer of Image Bind.

XAI

Elon Musk's AI company, mentioned in relation to Flux 1 being their image generation model before they developed their own.

Deepgram

A company providing voice infrastructure for AI agents, offering speech-to-text, text-to-speech, and voice agent APIs. Recently acquired Pipcat and is a sponsor of the podcast.

Pipcat

A company recently acquired by Deepgram, also a former sponsor of AIE.

More from Latent Space

View all 281 summaries

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free