Key Moments
Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club
Y CombinatorWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI can now write high-performance multi-GPU kernels and even outperform human experts, but debugging and verifying their correctness remains a significant challenge.
Key Insights
Networking can consume up to 50% of total runtime for workloads like llama MDB prefill, highlighting it as a major remaining bottleneck after single-GPU optimizations.
A significant portion of AI queries, up to 88.7%, could be routed to local accelerators running open-source AI, with intelligence per watt improving 3x and intelligence per joule improving 18x in under two years.
AI-generated GPU kernels are achieving competitive results, with some users achieving top rankings on kernel programming platforms without prior GPU programming experience, often submitting hundreds of lines of AI-generated code.
Batching simulators on GPUs can achieve millions of frames per second of experience, demonstrating over a hundredfold speedup compared to CPU-based implementations for reinforcement learning environments.
Specializing hardware for different inference phases, such as prefill or decode, can improve total cost of ownership (TCO) and efficiency, particularly for longer sequences or specific use cases.
Existing game engine design patterns, like Entity Component System (ECS), can be adapted to GPUs to handle complex gameplay logic and dynamic memory allocation, enabling high-throughput simulation environments.
The growing need for multi-GPU kernel specialization
The ML systems community has significantly improved single-GPU efficiency through techniques like Flash Attention and linear attention models. However, GPU networking has emerged as a major bottleneck, potentially consuming up to 50% of total runtime for certain workloads. This necessitates advancements in multi-GPU kernel design. Hardware is evolving rapidly with features like in-network compute and asynchronous bulk device-initiated networking, offering new opportunities. The challenge lies in writing kernels that can achieve fine-grained overlap of compute and communication, exploit new hardware capabilities, and remain simple to maintain. Current alternatives, such as off-the-shelf libraries like NCCL or compiler-based approaches, often fall short, leading to suboptimal performance or complex, hard-to-maintain code. The focus is shifting towards identifying core principles and trade-offs to simplify the design space and build minimal programming abstractions for efficient multi-GPU kernels.
Parallel Kittens: Simplifying multi-GPU kernel development
Parallel Kittens (PK) is a CUDA framework designed to simplify the creation of efficient multi-GPU AI kernels. It is built upon three key trade-offs identified in multi-GPU kernel design: transfer mechanism, scheduling of inter-GPU communication with intra-GPU work, and design overhead. PK offers a data structure for each memory hierarchy layer (registers, shared memory, global memory, and remote HBM) and communication primitives that leverage the most efficient transfer mechanism for a given task. It also provides program templates for different scheduling strategies. PK has demonstrated effectiveness by matching or surpassing hand-optimized kernels with significantly less code (50-100 lines vs. hundreds or thousands). Major AI companies, including Cursor and Together AI, are adopting PK for training and inference workloads on tens of thousands of GPUs.
Intelligence per Watt: Shifting towards local and efficient inference
The current era of AI, characterized by large data centers filled with TPUs and GPUs, is akin to the mainframe era of computing. While demand is soaring, many AI requests do not require frontier-level intelligence. This has led to a parallel trend of smaller, capable open-source models and increasingly powerful local accelerators (e.g., on Macs, consumer GPUs). A study on 'intelligence per watt' found that up to 88.7% of AI queries could be routed to local accelerators with open-source models. Over the past two years, intelligence per watt has seen a 3x improvement, and intelligence per joule has seen an 18x improvement, driven by better local models and accelerators. This shift suggests significant energy, compute, and cost savings, potentially redistributing inference demand away from centralized cloud infrastructure. While local accelerators are improving, they still lag behind specialized data center hardware like NVIDIA's B200, presenting an investment opportunity for further development.
AI's surprising proficiency in writing GPU kernels
AI systems are demonstrating remarkable capability in generating high-performance GPU kernels, even challenging established programming models and human experts. Platforms like KernelBot, designed for benchmarking and competition, reveal that individuals with no prior GPU programming experience can achieve competitive results using AI-generated code. This includes complex tasks like QR decomposition, where AI submissions can be tens of thousands of lines long and implement sophisticated dispatchers for different input shapes and precision requirements. However, a significant challenge remains in verifying the correctness of these AI-generated kernels. AI can exploit loopholes in evaluation frameworks, leading to 'reward hacks' that mimic desired behavior without true correctness, similar to historical issues like 'Dieselgate'. This necessitates robust evaluation and detection mechanisms, potentially involving AI-driven cheat detectors, to ensure reliable performance.
Heterogeneous infrastructure for specialized inference workloads
Inference is a heterogeneous workload with different phases (prefill, decode, KV cache creation, speculative decoding) stressing compute, network, and memory bandwidths differently. This heterogeneity suggests that co-designing infrastructure with specialized hardware for different phases can be highly beneficial. For instance, separating prefill (compute-bound) and decode (memory-bound) can allow for independent scaling. Disaggregating decode onto specialized systems, like SRAM machines offering high bandwidth and low latency, can improve Total Cost of Ownership (TCO) for longer sequences. Similarly, offloading memory-bandwidth-intensive parts of MLPs to SRAM machines while running compute-bound attention on GPUs can extend interactivity. Speculative decoding can also benefit from running the larger 'drafter' model on a more capable system than the 'verifier'. This full-stack co-design approach, considering data center power, cooling, networking, and performance modeling, is crucial for optimizing heterogeneous infrastructure.
GPU-accelerated game engines for reinforcement learning
Existing game engines, while great for simulating environments, are often inefficient for the throughput-oriented training demands of reinforcement learning (RL). Running thousands of copies of a game engine on CPUs creates bottlenecks. The solution lies in 'batch simulators,' where a single game engine simulates a batch of environments simultaneously on the GPU, achieving millions of frames per second. The challenge is that traditional gameplay logic, with its branchy code and dynamic memory allocation, is a poor fit for GPU programming frameworks. However, by adapting the Entity Component System (ECS) design patterns common in the game industry to the GPU, this can be overcome. ECS structures data into entities and components, stored in unified column stores in GPU memory, allowing for efficient, throughput-oriented dynamic memory allocation and easy procedural generation. Systems then operate on these components, enabling massive parallelism with minimal synchronization overhead. This approach has demonstrated over a hundredfold speedup compared to CPU baselines, making complex simulations practical for data-hungry RL algorithms.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Concepts
●People Referenced
Common Questions
The video highlights a trend towards massive specialization at the chip level, with different hardware designs optimized for training data centers versus inference data centers.
Topics
Mentioned in this video
A platform for detecting cheating in kernel programming competitions.
A popular deep learning framework, discussed as a reference for correctness and as a platform with evolving programming models.
A closed-source library by NVIDIA for high-performance matrix multiplication kernels.
A machine learning system used by Curser for training models on tens of thousands of GPUs, which adopted Parallel Kittens.
A parallel computing platform and programming model developed by NVIDIA, discussed in the context of kernel development.
Mentioned as one of the local open source AI models tested in the intelligence per watt study.
A platform for competitive GPU kernel programming, used to benchmark libraries and track popular approaches.
Mentioned as one of the local open source AI models tested in the intelligence per watt study.
A family of open source models mentioned in the context of testing local AI models and accelerators.
A follow-up project focused on operationalizing insights from the intelligence per watt study, enabling on-device personal AI.
Peripheral Component Interconnect Express, the interface for connecting CPUs and GPUs, noted as a potential bottleneck.
Tensor Memory Accelerators, a transfer mechanism for sending data over NVLink, offering good throughput regardless of message size.
Mentioned as one of the local open source AI models tested in the intelligence per watt study.
A high-speed interconnect technology developed by NVIDIA that allows direct communication between GPUs.
A programming language for GPU kernels, mentioned as an alternative to Triton, with contributions from Stuart.
A CUDA framework designed to simplify and accelerate multi-GPU AI kernels, built on identified trade-offs.
A library for collective communication operations, mentioned as an off-the-shelf option with coarse-grained primitives.
A closed-source library by NVIDIA, similar to Cutlass, for high-performance matrix operations.
A startup focused on building workload-optimized heterogeneous infrastructure for inference.
Mentioned indirectly as a provider of CPUs, contrasting with GPU efficiency.
Mentioned as a provider of accelerators for personal consumer use, contributing to the trend of local AI inference.
A major provider of GPUs and AI infrastructure, mentioned in the context of training and inference hardware, and their role in high-performance computing.
A robotics company founded by Trevor Blackwell, mentioned in the context of YC's founding story.
A company co-founded by Mark and Casey Elward, involved in custom kernels and competitive GPU programming.
Mentioned as an example of 'dieselgate', a real-world instance of an entity cheating on emissions testing, analogous to reward hacking in AI.
Mentioned in the context of historical mainframes that filled entire rooms, drawing a parallel to current data center scale.
Mentioned in the context of their hide-and-seek environment and as a provider of AI services that can be replaced by local inference.
Mentioned as a provider of TPUs for cloud-based AI training and inference.
Mentioned as a company specializing in decode engines for inference, indicating a split in specialization.
A company co-founded by Mark and Jerry Torque, focusing on AI and automation.
Mentioned as a provider of GPUs for personal consumer use, contributing to the trend of local AI inference.
A GPU used for demonstrating the performance of the GPU-accelerated game engine, simulating 4,000 hide-and-seek environments.
An NVIDIA GPU architecture mentioned in the context of its SM count and as a driver for newer programming models like Triton.
High Bandwidth Memory, the main memory system for modern GPUs, crucial for kernel performance.
Mentioned as an example of early chip specialization in training data centers.
A local accelerator from Apple, mentioned as lagging behind data center GPUs like NVIDIA B200 in intelligence per watt and jewel.
A data center GPU mentioned as a benchmark against which local accelerators like the Apple M4 Max are compared for intelligence efficiency.
A specialized accelerator for inference, noted as being significantly ahead of consumer accelerators in performance.
An algorithm that is computationally intensive in PyTorch, mentioned in the context of optimizing it for use with the Shampoo optimizer.
An optimizer that relies on QR factorization, mentioned as a reason for slow deployment due to performance limitations in PyTorch.
A design pattern used in game development for managing entities, components, and systems, adapted for efficient GPU execution in simulations.
More from Y Combinator
View all 616 summaries
31 minPatrick Collison: "What If You Succeed?"
58 minJeff Dean: The 1% Rule for Building in AI
33 minAlexandr Wang: “This is a Once-in-a-Civilization Opportunity”
51 minBlake Scholl: The Problem With Becoming an Expert
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free