Key Moments
Going In Deep On Data | YC Paper Club
Y CombinatorWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Data is the primary bottleneck in AI development, not architecture or compute, and creating high-quality, specialized datasets requires significant craftsmanship and expertise, moving beyond simple labeling.
Key Insights
The market cap creation for data businesses has surged to over $100 billion in the last 10 years, drastically shifting VC sentiment from data being a commodity to a critical asset.
Manual labeling is not scalable, robust to noise, or adaptable to spec changes, leading to the development of 'data programming' and weak supervision techniques to encode expert knowledge programmatically.
New benchmarks like Senior SWEBench are being developed to evaluate AI agents on tasks representative of senior engineers, moving beyond junior-level evaluations and requiring nuanced reward design.
Diffusion language models can achieve speeds over 1,000 tokens per second, significantly outperforming traditional autoregressive models and enabling real-time AI applications, especially in voice.
Multilingual pre-training reveals significant language synergies and interferences, where the amount of data for a language like Thai is less than 0.6% of English data, impacting model performance.
Data 2.0 requires more sophisticated data research than simply adding more human labels, focusing on thoughtful design that leverages expert supervision and judgment in complex, dynamic environments.
Data's transformation from commodity to critical asset
The perception of data in AI development has dramatically shifted. Initially, around 2016, data was considered a commodity, with little perceived value beyond readily available datasets like ImageNet. VCs largely dismissed the terminal value of data businesses, anticipating a market cap of zero. However, this perspective has fundamentally changed. Over the past decade, data-centric AI has driven over $100 billion in market cap creation, making it one of Y Combinator's most successful investment categories. The core reason for this shift is the realization that data quality and its effective use are paramount. When training models, especially for complex real-world applications, the correct approach to improving performance is not solely through architectural changes or adding more layers, but by meticulously examining the data. Analyzing false positives and negatives, identifying failure modes (e.g., fogged-up refrigerators or obscured products in retail), and using this analysis to refine the dataset is crucial. This iterative process, driven by understanding the data, proves far more impactful than purely architectural innovation, especially with expressive models like transformers.
The limitations of manual labeling and the rise of data programming
Manual labeling, the traditional approach to dataset creation (Data 1.0), suffers from several critical drawbacks. It is inherently not scalable, with each data point requiring individual human effort, making it an O(N) problem. Furthermore, it is not robust to noise; reducing noise necessitates extensive redundancy (e.g., K by N relabeling), increasing costs and effort. Changes in task definitions or specifications require starting the labeling process from scratch, and there is often no clear provenance or rationale behind the labels. This becomes particularly intractable in expert domains like medicine or niche scientific fields, where the cost and scarcity of expertise make manual labeling for large datasets of complex data (e.g., MRIs, EHRs) virtually impossible. To address this, the concept of 'data programming' emerged. This approach encodes expert supervision into software, leveraging the scalability and adaptability of code. Instead of manual labels, expertise is captured through heuristics or 'labeling functions.' These functions represent expert heuristics and reasoning, making the process reproducible and scalable. The challenge of overlapping and potentially inaccurate signals from these functions is managed through 'weak supervision' techniques, which model the accuracy of each source in an unsupervised manner. A 'label model' then denoises these signals to create a higher-quality dataset for training, effectively enabling programmatic data labeling.
Scaling expertise for complex data 2.0 challenges
As AI systems become more sophisticated, moving into Data 2.0, the complexity of data requirements escalates significantly. This is exemplified in domains like coding, where agents are expected to handle not just simple Q&A but complex tasks involving intricate prompts, dynamic environments, and extended sequence lengths. While simple benchmarks like HumanEval might be saturated, new challenges arise with terminal-based agents and more complex benchmarks like Senior SWEBench. This benchmark aims to evaluate AI agents as senior engineers, capable of architectural decisions and code refactoring, rather than just junior developers. Creating such benchmarks requires extensive expert supervision. The design process for these tasks involves natural language instructions, more realistic higher-level abstractions (e.g., Slack messages, log dumps), and sophisticated reward mechanisms that capture aspects like 'taste' and alignment with code base practices, not just correctness. The challenge is scaling this expertise. Snorkel's approach involves a 'validation agent' that bridges the gap between traditional deterministic verifiers (high reliability, low flexibility) and LLM judges (low reliability, high flexibility). This validation agent translates user stories and specifications from experts into deterministic test scripts that execute against the code, capturing expert intuition in a scalable and reliable manner, ultimately leading to metrics like 'tasteful pass' to assess nuanced code quality.
Diffusion models enable real-time AI with high throughput
A new generation of language models, powered by diffusion, offers a significant leap in inference speed. Unlike traditional autoregressive models that generate tokens sequentially, diffusion models generate tokens in parallel, starting from an initial guess and refining it over multiple steps. This parallel processing allows for significantly higher throughput, reaching speeds of over 1,000 tokens per second. This breakthrough has profound implications for real-time AI applications, particularly in voice. In cascaded voice pipelines (speech-to-text, LLM, text-to-speech), the LLM is often the latency bottleneck. By using diffusion models like Inception's Mercury 2, which achieve high speeds and quality, the overall latency can be dramatically reduced. This enables more seamless real-time voice interactions, allows for larger or more capable models to be used within latency constraints, or permits longer reasoning for improved quality. These diffusion models can achieve a new Pareto frontier of quality and latency, even outperforming specialized hardware designed for autoregressive models when run on more accessible GPUs, offering a software-based solution for speed.
Multilingual pre-training: Synergy, interference, and data scarcity
Training large language models in multiple languages presents unique challenges, particularly concerning data availability and cross-lingual transfer. For instance, in common crawl-based corpora like Madlad 400, the amount of data for languages like Thai is less than 0.6% of that for English. Training a model solely on scarce data leads to overfitting and poor performance. Conversely, including such low-resource languages in massive multilingual models means they receive a tiny fraction of the training mixture, again resulting in suboptimal performance for speakers of those languages. Research in multilingual pre-training has focused on quantifying language synergies and interferences. By comparing training curves of monolingual models with multilingual ones, researchers can measure how much one language helps or harms another. This analysis reveals a complex 'cross-lingual transfer matrix' where some languages exhibit high synergy (e.g., Portuguese and Italian for Spanish), while others show significant interference (e.g., Japanese for Spanish). Interestingly, these relationships are not always symmetric, and the size of the model plays a crucial role: larger models can accommodate more synergies, while smaller models are more prone to interference. Script similarity also appears to be a notable factor, alongside language family, in facilitating transfer.
Data Forge: Synthesizing realistic RL environments for evaluation and training
To address the limitations of existing benchmarks like ToBenc h, which may not capture the full complexity of real-world scenarios, approaches like Inception's 'Data Forge' are being developed. Data Forge is an agentic system that synthesizes a wide range of realistic Reinforcement Learning (RL) environments. It uses real-world data from user interactions or partnerships with companies to create environments that are representative of actual usage. The process involves synthesizing an initial environment based on domain descriptions and real-world data, creating policies for domain operation, and defining tools for agents to use. Crucially, it generates tasks by defining abstract scenarios and user personas, then specializing them to the target domain. This generation process involves an iterative procedure to 'harden' tasks, balancing difficulty to provide sufficient learning signal without being overwhelmingly hard. This ensures that the synthesized environments and tasks closely mirror real-world interactions, providing valuable data for both training and evaluating models, especially for latency-sensitive applications like voice agents in domains such as banking or travel.
The future of data is expert-driven and benchmark-focused
The trajectory of AI development clearly indicates that data, not just compute or architecture, is the primary bottleneck and frontier. As we move into Data 2.0 and beyond, the complexity of environments, the nuance and subjectivity of outputs, and the autonomy of agents will continue to grow. This necessitates a shift from simply labeling data to deeply understanding and engineering it. Future progress will rely on thoughtful data design that leverages expert supervision and judgment. There is a strong call for more sophisticated benchmarks that accurately reflect real-world complexity and agent capabilities. Initiatives like YC's 'Open Benchmarks Grants' aim to accelerate research in this area, supporting the development of new benchmarks across various domains. The focus is on enabling experts—software engineers, doctors, lawyers—to exercise their supervision and judgment within these data sets and evaluation frameworks, making data development a critical research problem in itself.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
●People Referenced
Cross-Lingual Transfer Matrix (Synergy/Interference)
Data extracted from this episode
| Target Language | Source Language | Transfer Effect (Red: Synergy, Blue: Interference) |
|---|---|---|
| Spanish | Portuguese | High Synergy (Red) |
| Spanish | Italian | High Synergy (Red) |
| Spanish | French | High Synergy (Red) |
| Spanish | Japanese | High Interference (Blue) |
Common Questions
As models become more expressive, the nuances and real-world distribution of data become the primary bottleneck. Simply increasing model complexity or layers doesn't solve issues like fogged-up camera lenses or occluded views, which require better data handling and analysis.
Topics
Mentioned in this video
University where Vololo is a professor.
The speaker mentions Andreessen Horowitz in the context of VCs and data businesses, implying they may have changed their tune on data's value.
Mentioned as a university from which PhDs were interviewed, and where Snorkel originated its research.
Mentioned as a university from which PhDs were interviewed.
The origins of Snorkel's research in data programming date back to this lab.
The institution from which the original SWEBench team collaborated on the senior SWEBench project.
Mentioned as a university with benchmark projects like agents/exam.
The organization hosting the paper club event, offering credits to startups.
A company started by the speaker in 2016, which focused on data-driven AI development.
Used as an analogy for how LLMs could be specialized, similar to how different apps serve specific functions on a phone.
Used as an analogy to illustrate that specialized companies are often better than large tech firms at niche services, even if the large firms could technically build them.
Mentioned as a company that, along with others like OpenAI and Google, might benefit from specialized data companies rather than trying to build all specialized AI services themselves.
Mentioned as a company that, along with Anthropic and Google, might benefit from specialized data companies.
Mentioned as a company that, along with Anthropic and OpenAI, might benefit from specialized data companies.
A company founded by Vololo, focused on training large-scale diffusion language models.
An evaluation framework mentioned in the context of building benchmarks for senior software engineers.
A company whose specialized chips were compared to GPUs for running aggressive models.
A preview model from Inception Labs that showed significant improvement after training on tasks synthesized from Dowo Forge.
Venture capital firm where Levy worked, who recognized the value of scale series A funding.
An AI architecture that the speaker notes works well and has shifted the focus of AI development towards data.
A company focused on data and scaling expertise, with roots in Stanford AI Lab. Vincent Chen is a founding team member.
Environments used to represent complex tasks and rubrics for AI agents.
A language model that tied for first place in the senior SWEBench benchmark.
A language model that tied for first place in the senior SWEBench benchmark.
A language model that tied for first place in the senior SWEBench benchmark.
A system developed by Inception Labs to synthesize realistic RL environments for training and evaluating language models, based on real-world data.
A benchmark for coding agents, the original SWEBench team from Princeton collaborated on senior SWEBench.
A benchmark for evaluating language models, particularly in voice applications. The speaker points out its limitations in capturing real-world complexity and its tendency to be 'benchmarked out'.
A scaling law paper that Shane's team uses as a basis for their multilingual scaling law estimations.
More from Y Combinator
View all 627 summaries
40 minMichael Kratsios: Inside the White House's AI Strategy
52 minSusan Kare: Designing Icons & Graphics For the Original Mac
59 minChelsea Finn: This is the State of the Art in Robotics
42 minPeter Steinberger: What Happens When 4.7 Million People Let It Cook
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free