Key Moments
Why Inference Matters 12x More Than Training | YC Paper Club
Y CombinatorKey Moments
Inference consumes 6x more FLOPs than training annually, costing $220B market cap over 10 years; optimizing it is key for AI profitability.
Key Insights
For every FLOP spent training a model, roughly six go to serving it each year, escalating its importance by 12x over a model's two-year lifespan.
DeepSeek's 80% profit margin is attributed to significant innovations in inference, highlighting its financial impact.
GPU utilization during inference is drastically low: 50% during prefill and as low as 5% during decoding, showcasing the need for optimization.
The throughput difference between prefill (1.5M tokens/sec) and decoding (14,000 tokens/sec) reveals decoding as the true bottleneck.
Inference routers must balance load across data centers and GPUs while considering KV cache locality for optimal performance.
New hardware, like NVIDIA's 72-GPU racks, enables massive scaling for Mixture of Experts (MoE) models, increasing computational density.
Inference's overwhelming cost and market impact
The critical importance of inference, the process of using a trained AI model, is often overshadowed by the focus on training. However, inference consumes significantly more computational resources. It's estimated that for every FLOP spent on training, approximately six FLOPs are spent on serving the model annually. Over the last decade, this focus on inference has unlocked over $220 billion in market capitalization. DeepSeek, a company highlighted for its profitability, achieved an impressive 80% profit margin largely due to their innovations in inference efficiency. This underscores that while training builds the model, inference is where the economic value is largely realized and sustained.
The 12x importance of inference over training's lifespan
The financial and computational significance of inference escalates dramatically when considering the entire lifecycle of a model. Data from DeepSeek suggests that inference costs can be six times higher than training costs on an annual basis. Factoring in a typical model lifespan of two years, inference becomes twelve times more important than training. As AI models become more ubiquitous and their usage grows, this disparity will only widen. This perspective shifts the research and development focus from solely minimizing training costs to a more balanced approach that prioritizes inference efficiency, which is crucial for sustainable AI deployment and profitability.
Understanding the complexity of the inference stack
Contrary to the simple notion of calling 'model.forward()', inference involves a complex stack of operations. This includes sophisticated routing, caching, scheduling, kernel execution, memory management, and hardware utilization. Each token processed must navigate this intricate pipeline. For Large Language Models (LLMs), the process begins with a prompt, which is tokenized. The 'prefill' phase processes this initial prompt. If parts of the prompt have been seen before (cached), they are not reprocessed. Following this is the 'decoding' phase, where the model generates tokens one by one. This phase is significantly more computationally intensive and memory-bound than prefill. Optimizing each stage, from efficient caching strategies (like LRU) to kernel execution (e.g., FP8 precision), is vital for reducing latency and cost.
The staggering underutilization of GPUs during inference
A significant challenge in inference is the extremely low utilization of GPU compute. While training and prefill phases might achieve around 50% GPU utilization using formats like FP8, the decoding phase, which generates subsequent tokens, often operates at a mere 5% utilization. This means that a vast majority of the computational power of expensive GPUs is going to waste during the most critical part of the inference process. Improving this drastically low utilization is paramount for making large-scale AI inference economically viable and performant.
Key performance indicators in inference: Speed and throughput
Researchers and engineers focus on several key metrics to optimize inference. The 'time to first token' (TTFT) measures how long it takes to complete the prefill stage and generate the very first output token. 'Time per output token' (TPOT) measures the latency for each subsequent token generated during decoding. 'Throughput' quantifies the number of tokens generated per unit of time, often expressed as tokens per second per GPU. High throughput is essential for serving many users concurrently. For instance, with efficient KV caching, a data center might achieve 1.5 million tokens per second per GPU during prefill, but only 14,000 tokens per second during decoding, highlighting the bottleneck.
KV caching strategies and computational density
Key-Value (KV) caching plays a crucial role in speeding up inference by storing intermediate states of the attention mechanism. Companies like DeepSeek reportedly manage petabytes of user KV cache data, retaining it for up to 72 hours, which is essential for high cache hit rates. Another critical concept is 'computational density,' which measures how many computations can be performed for a given amount of data moved between memory (like DRAM) and compute units (like SRAM). For traditional computer vision tasks, computational density can be very high. However, for LLMs, it is often very low, meaning more data needs to be moved relative to the computations performed, exacerbating memory-bound bottlenecks. Innovations aim to increase this density, either by bringing computation closer to memory or by optimizing data movement.
Model, kernel, and memory optimization techniques
Innovations in inference optimization fall into three categories: model architecture, kernel/runtime, and memory harnessing. Model innovations include techniques like Mixture of Experts (MoE) which selectively activate parts of a model, or State Space Models (SSMs) that reduce the need for full attention in every layer. Kernel and runtime innovations draw inspiration from classic operating systems and database theory, focusing on efficient implementations like FlashAttention or PagedAttention. Memory harnessing encompasses intelligent routing and speculative decoding. Quantization, reducing the precision of model weights (e.g., from BF16 to 8-bit or 4-bit), is a vital technique for reducing model size and memory footprint, though it requires careful quality assurance to maintain performance.
The role of specialized hardware and parallelization
The advancement of specialized AI hardware, like NVIDIA's Blackwell GPUs and their associated high-bandwidth interconnects (e.g., connecting 72 GPUs), is enabling massive scaling. This is particularly beneficial for MoE models, where different 'experts' can be distributed across many GPUs. Traditional parallelization schemes (tensor, pipeline, data) are being enhanced by techniques like expert parallelism. However, managing reliability in such large-scale distributed systems is a significant challenge, as a single node failure can impact a large portion of the cluster. Building fault-tolerant systems is crucial for maintaining continuous inference service.
The challenges of open-source model inference and quality assurance
While open-source models offer cost and flexibility advantages, a significant challenge lies in the variability of inference quality across different providers. Unlike closed-source models which offer a consistent API, open-source models can exhibit performance gaps between the best and worst providers that are larger than the gap between open and closed models themselves. This inconsistency affects crucial aspects like accuracy, speed, tool-calling effectiveness, and even subtle issues like incorrect model names or silent service updates. Robust quality assurance, including continuous evaluation, benchmarking, and specialized test suites, is vital for developers building applications on open-source models to ensure a reliable user experience.
Compilers and the quest for efficient model deployment
The development of specialized compilers, like Luminal's, is critical for efficiently translating AI models into executable code for diverse hardware. These compilers abstract away hardware complexities by reducing models to a core set of operations and using techniques like 'superoptimization' to explore vast search spaces for optimal code generation. The challenge lies in balancing compilation time with runtime performance. For inference, where models are deployed and run millions of times, a longer compilation time is acceptable if it yields even marginal improvements in inference speed and cost-efficiency. The goal is to democratize access to high-performance AI hardware through sophisticated software.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
DeepSeek Inference vs. Training Cost
Data extracted from this episode
| Metric | Value |
|---|---|
| Training hours (H1800 for DeepSeek V3) | 2.8 million |
| Equivalent H800 hours/day | 43,000 |
| Inference importance (annual) | 6x training |
| Inference importance (2-year lifespan) | 12x training |
| DeepSeek Profit Margins (from inference innovations) | 80% |
Inference Throughput and Latency Metrics
Data extracted from this episode
| Metric | Value |
|---|---|
| Prefill Throughput (per GPU, excluding cache hits) | 60,000 tokens/sec |
| KV Cache Hit Rate | 96% |
| Prefill Throughput (per GPU, with cache hits) | 1.5 million tokens/sec |
| First Token Latency (P50) | 400 ms |
| First Token Latency (P99) | 1 sec |
| Decoding Throughput (per GPU) | 14,000 tokens/sec |
Common Questions
Inference scales significantly more than training. DeepSeek's data suggests that inference operations are six times greater than training annually, and considering a two-year model lifespan, this importance increases to twelve times, especially as AI adoption grows.
Topics
Mentioned in this video
A company mentioned for publishing its training and inference data, and for having high profit margins (80%) due to inference innovations.
Open-source model provider Together AI collaborated with to quantize their M3 model to NVFP4, matching original performance.
Mentioned as a leading AI company, with Together AI processing more tokens than OpenAI for a day, and for using recurrent models.
Mentioned as the company behind Gemini, with Together AI achieving a significant fraction of its traffic volume.
A specialized AI cloud platform focusing on inference, processing 400 trillion tokens monthly and aiming to provide an AWS-like experience for AI companies.
GPU manufacturer, whose Blackwell chips enable high-bandwidth communication between up to 72 GPUs, a significant advancement for model parallelization.
Mentioned as an AI company that also uses recurrent models and experienced a quality outage due to a small TPU kernel bug.
A company specializing in compilers, aiming to define the transformation function between GPU FLOPs/hours and tokens, making them profitable.
A team that released the Gemini K2 tool checker to encourage competition in tool use accuracy for Gemini models.
The primary developer of XLA and Miles, contributing to open-source AI infrastructure.
An inference engine that Luminal's compiler works within, leveraging its solutions for speculative decoding.
An internal implementation where SGLang manages agent memory as a root tree.
A model trained by DeepSeek, requiring 2.8 million H1800 hours for training, highlighting the scale of LLM training.
An AI model used as a benchmark for comparison with Llama models, with some Llama endpoints performing similarly.
A framework that powers Ollama on Apple Silicon, enabling local model inference.
NVIDIA's matrix multiplication library, used as an example of a host-level operation that can be invoked from the CPU.
An inference engine that Luminal's compiler works within, leveraging its solutions for speculative decoding.
A component primarily developed by Red Hat Arc, responsible for deployments and the ideal backend for Miles RL training.
An example of an LLM where users frequently start new, unrelated conversations, impacting KV cache efficiency.
An open-source project launched in 2023 to run open models locally, now also offering a cloud service for larger models.
The engine powering Luminal's equality saturation, which allows for robust error debugging by providing a proof of rewrite sequences.
Another NVIDIA library, providing thread-block level operations for GPU kernel programming.
A library from Hazy Research that implements thread-block level operations.
An internal Together AI project using agents to automatically optimize kernel performance for target models, devices, and data sizes.
Google's AI model, whose traffic volume was compared to Together AI's, and whose tool checker was launched by Moonshot team.
A family of open-source models, initially smaller and runnable locally, later tested by a 2024 paper showing significant performance variance across different endpoints.
Compared to Together AI's vision as a specialized AI cloud, offering CPU, compute resources, databases, storage, and REST APIs.
A model mentioned for running Qwen 3.5 MoE locally from disk.
Traditional compiler stack, contrasted with Luminal's unique approach using equality saturation.
An AI agent mentioned as a potential technology for learned-guided search in compiler optimization.
A new local model, launched with Ollama as a partner, noted for its excellent performance in local programming tasks.
An inference system developed by Red Hat Arc, supporting fully asynchronous execution for continuous generation and training.
An AI model, specifically a Mixture-of-Experts (MoE) model, that can be run locally by Colibri.
NVIDIA's software ecosystem, recognized as a major contributor to NVIDIA's valuation due to its excellent support for complex hardware.
A deep learning framework mentioned for its extensive set of operations (around 2000), which Luminal's compiler can ingest and compile.
MiniMax's model, successfully quantized to NVFP4 by Together AI, achieving performance matching the original FP8 model.
NVIDIA GPU architecture, noted for its ability to connect 72 GPUs with higher bandwidth, improving parallelization for large models.
Hardware platform where Ollama with MLX can run models locally.
NVIDIA's Grace Blackwell chips enable highly interconnected GPU clusters beyond the previous 8-GPU limit, allowing for better model parallelization.
CEO of NVIDIA, mentioned in the context of GPU performance guarantees and actual achievable FLOPS.
Co-founder of You.com and AI researcher, mentioned in the context of launching recurrent models.
Head of Research at Together AI and former Hazy Lab member, presenting on token lifecycle and inference optimization.
Co-founder and CEO of Luminal, a company specializing in compilers for efficient AI inference.
Co-founder and CEO of Ollama, presenting on the differences and quality measurement of open models.
Head of SGLang and Really Stark, discussing agent workload optimization and memory management.
More from Y Combinator
View all 639 summaries
48 minThis Is How You Should Actually Make Launch Videos
48 minHow To Stand Out In A World Of AI Designs
98 minAI Safety Fact vs. Fiction | YC Paper Club
44 minHow Jeetu Patel Runs Cisco Like the World’s Largest Startup
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free