Key Moments
The Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI models can now write their own optimized GPU kernels, but achieving peak inference performance involves complex, multi-layered engineering beyond just model weights.
Key Insights
GLM-5.2 can analyze performance profiles, identify bottleneck kernels, and rewrite them for optimization, with some of these kernels then being written by GLM-5.2 itself.
Speculative decoding, a technique to speed up token generation by having a smaller model predict multiple tokens at once, relies on traffic-specific training to maintain high acceptance rates.
Combining components from different open-source models, like a vision encoder from one with LLM weights from another, is a key strength of the open-source ecosystem.
Quantization, while a lossy optimization that improves speed, can sometimes lead to better fidelity if quantization errors are carefully managed and cancel each other out, as demonstrated by a research intern's work showing 20% more quantized models with better quality.
Inference engineering for data centers focuses on reducing latency (making models faster), while local AI inference engineering prioritizes fitting models onto hardware and reducing their 'dumbness' through quantization, pruning, and distillation.
Video diffusion models face significant challenges with attention bottlenecks due to the sheer number of tokens required, leading to a trade-off between full attention (high quality, infeasible compute) and sparse attention (lower quality) or an exploration of autoregressive video generation.
AI models writing their own optimization code
A fascinating demonstration of AI self-improvement involves models like GLM-5.2 being used to analyze their own performance. By plugging the model into a cloud code harness, it can perform forward passes, capture profile traces, identify bottleneck GPU kernels, and then rewrite those kernels for optimization. This iterative process, where the model optimizes itself by generating and testing new code, showcases a remarkable level of autonomy in performance tuning. Some GPU kernels within Baseten's inference engine have, in fact, been written by GLM-5.2 itself.
Optimizing inference speed with speculative decoding and cache-aware routing
When processing a long query (e.g., 200,000 tokens), the first consideration is whether parts of the query have been seen before, leveraging cache-aware routing to send requests to instances with available prefill workers and cached inputs. This significantly reduces the computational load by skipping the prefill step for cached tokens. For new inputs, some models employ disaggregated prefill and decode stages, with separate GPU sets handling input processing and token generation. Speculative decoding further enhances speed. A smaller, 'speculator' model rapidly generates several potential tokens, which are then verified by the main model. This drastically speeds up decode if the speculator has a high acceptance rate. However, the effectiveness of speculative decoding is highly traffic-specific; a speculator trained on coding tasks will perform poorly if the user is summarizing Harry Potter books. For dedicated deployments, custom speculative models can be trained, offering significant speedups tailored to specific use cases.
The complexities of production-ready model deployment
Moving from a model that can simply 'make a token' to a production-ready API involves substantial engineering. This includes re-performing quantization to ensure compatibility and optimal performance (e.g., using NVFP4 for Blackwell compatibility), and meticulously calibrating these quantized weights to avoid regressions in model intelligence. Training speculative decoders is crucial, leveraging public datasets representative of popular use cases like coding and agentic tasks. The process requires the actual model weights to extract hidden states for training the speculator. Furthermore, adapting to new model architectures, such as the sparse attention mechanisms seen in models like DeepSeek or the vision capabilities grafted onto GLM-5.2 by combining components from different models (e.g., a vision encoder from Kimi with GLM weights), adds layers of complexity. Each new model architecture can introduce novel challenges that require runtime support and careful integration.
Structured output and the challenges of tool calling
Tool calling, while seemingly straightforward, presents significant challenges, especially when customers require highly specific, post-trained models. The core issue often lies not in the sandboxing or containment of the LLM, but in its ability to correctly parse and utilize the JSON outputs required for tool invocation. If the post-training or quantization process is suboptimal, the model may struggle to reliably read JSON. A common failure mode is the model performing the 'thinking' and 'tool calling' but then hallucinating the result during decoding because it didn't properly see the complete response. To mitigate this, techniques like structured output generation, using state machines or BNF grammars to constrain the output format, are employed. This ensures the output adheres to a specific structure, though it doesn't guarantee the model will call the correct tool or avoid certainty issues.
Quantization: a balance of speed and fidelity
Quantization, a primary optimization technique, reduces model size and increases inference speed by using lower-precision data formats (e.g., FP16 to INT4). While technically a lossy process, the goal is to preserve model fidelity as closely as possible to the original full-precision model. Research has shown that it's not always a case of 'more quantization equals worse quality.' In some instances, by strategically quantizing specific layers and calibrating the weights, quantization errors can effectively cancel each other out, leading to a model that is more heavily quantized (e.g., 20% more) yet retains or even improves fidelity compared to less quantized models. This is assessed not just by standard benchmarks but also by comparing logit distributions using metrics like KL divergence to ensure the quantized model's output probabilities closely match the original.
The evolving landscape of AI hardware and parallelism
The field is seeing rapid advancements in AI hardware, with new generations like NVIDIA's Blackwell and the upcoming Rubin pushing boundaries in memory bandwidth and compute power. While tensor parallelism (sharding a model across GPUs) is beneficial for latency and requires high-bandwidth interconnects like NVLink, expert parallelism (placing entire 'experts' of a Mixture-of-Experts model on different GPUs) can increase throughput with less critical inter-GPU communication. Pipeline parallelism is typically reserved for multi-node inference when a model exceeds single-node capacity. The memory requirements for large models, especially with long context lengths, are immense; for instance, a 2.8 trillion parameter model in FP4 requires approximately 1.4 terabytes, necessitating substantial GPU VRAM (e.g., 288 GB per GB300 GPU) and efficient KV cache management. The trend suggests GPUs are becoming more specialized, incorporating tensor cores and instructions tailored for AI workloads, blurring the lines with ASICs.
Video generation's unique inference challenges
Inference engineering for video diffusion models presents distinct challenges compared to LLMs. Models are generally smaller, and batching is less common, with each request often served by a single GPU. However, the sheer number of tokens required for high-quality video (e.g., 16 frames per second at 480p) creates massive attention bottlenecks due to the quadratic complexity of full attention. This forces a choice between computationally infeasible full attention or sparse attention methods that compromise video quality. Autoregressive video generation offers a potential solution, allowing for streaming output and application of LLM-like optimizations, but current open-source models often suffer from degraded quality and frame-to-frame consistency issues, leading to 'drift' over longer sequences. While closed-source models use proprietary techniques to stitch video chunks effectively, the open-source community is still working towards robust autoregressive video generation or significant breakthroughs in efficient full attention.
The convergence of training and inference engineering
The lines between training and inference engineering are increasingly blurring. Inference optimizations are becoming critical for training pipelines, particularly for efficient rollouts and data generation. Conversely, training techniques are vital for inference, especially in areas like post-training quantization, where models may need fine-tuning after quantization to regain performance. Techniques like quantization distillation, where an FP4 model is trained to mimic the logits of a full-precision model, highlight this convergence. The ultimate goal is a continuous improvement loop where models learn from live inference data, dynamically adjust systems, and are then retrained and rolled out. This unification accelerates the development of more capable and efficient AI systems, with models potentially even optimizing their own inference processes, as seen with GLM-5.2's ability to write GPU kernels.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Books
●Concepts
●People Referenced
Common Questions
Baseten first checks for cached inputs to skip prefill. If not cached, it disaggregates prefill and decode, using separate GPUs for each, and may employ a speculative model, especially for coding tasks, to iteratively generate tokens.
Topics
Mentioned in this video
Guest on the podcast, co-author of 'Inference Engineering' and affiliated with Baseten, discussing LLM inference.
Guest on the podcast, known as the 'Waterloo intern' and discussing various aspects of AI inference and model optimization.
Used as an example of a scientist that a retrofitted vision model might incorrectly identify from an image, yet still categorize correctly.
Used as an example for a vision model to identify, highlighting its ability to categorize a person as a 'scientist' even if the specific identity is wrong.
A finance professor whose paper on quantitative finance research was cited for illustrating the narrowing margins in optimized domains.
Founder of Midjourney, mentioned for his past investment in text diffusion, highlighting the idea of coherent long-form generation.
A speaker from a previous podcast, cited for his perspective on vertically integrated model lab ASICs for large training runs.
A researcher cited for work on inverse scaling laws.
A member of the Grock Imagine team who previously shared hints about their video generation approach.
An AI inference platform where the guests work, providing insights into their infrastructure and optimization strategies.
A platform for AI models, mentioned as a source for new model releases and associated support work.
An AI model, noted for its novel architectural innovations and attention mechanisms.
Mentioned as a vendor accused by Kimmy of not performing well on their benchmarks.
A major GPU manufacturer, whose images and hardware (like B200s) are central to LLM inference, and also a competitor in quantization optimization.
A text diffusion model mentioned as being hosted and a potential alternative API for text generation.
Mentioned in the context of creating custom ASICs for massive training runs.
Mentioned as a potential partner for OpenAI in developing custom ASICs.
Graphics Processing Unit, central to AI inference calculations, discussed in the context of kernel writing and processing queries.
A foundational model for vision-language tasks, mentioned in the context of building questions off images.
A type of Visual Language Model or open-source inference engine, mentioned in comparison to proprietary stacks.
An open-source inference engine, often receives weights ahead of time for new models.
A vision-language model, its paper described as foundational work for vision processing.
A model from AI21 that specifically designs its architecture dimensions for a target hardware to maximize saturation.
A future NVIDIA GPU, speculated to have architectural features that mitigate the need for mega kernels and emphasize system-level optimization.
A theoretical or future version of OpenAI's GPT model, used as a benchmark for open-source LLM capabilities.
A closed-source video generation model, chosen by media companies for higher quality despite higher cost.
An earlier version of Meta's open-source language model, highlighting model longevity and trust in enterprise settings.
NVIDIA's own models, known to run very well on Blackwell hardware, suggesting hardware-targeted training.
An image generation model, cited as an example of an auto-regressive image model.
An AI art generator, its founder David Holtz was mentioned for his interest in text diffusion.
NVIDIA's parallel computing platform and programming model, mentioned in the context of low-level GPU programming challenges.
A closed-source video generation model, noted for its higher quality compared to open-source alternatives.
An open-source language model, still seeing significant workloads despite newer models.
A new 'world model' type of AI, mentioned as an upcoming modality in AI research.
An image generation model, cited as an example of an auto-regressive image model.
An open-source diffusion model for text, noted for its performance relative to its size.
A large language model, specifically the 3-trillion parameter version, used as an example of a model that requires specialized hardware like GB300s.
OpenAI's foundational language model, referenced for its early streaming behavior where text was generated progressively.
A popular open-source machine learning framework, mentioned in the context of guaranteeing execution order in GPU operations.
A quantization format used to optimize models for maximum Blackwell compatibility and throughput.
Backus-Naur Form grammar, used to constrain model output to specific formats like JSON.
An older technique for speculative decoding, mentioned as a historical example in the book 'Inference Engineering'.
A newer technique in speculative decoding, mentioned as an evolution from older methods like Medusa.
An optimization technique that significantly improves memory bandwidth on local devices like MacBooks but is less effective on high-bandwidth GPUs like B200s.
An older architecture for audio processing, enabling auto-regressive speech generation by adding waveforms to the LLM's vocabulary.
An open-source library from NVIDIA for moving information around a GPU cluster, described as a developer toolkit rather than an out-of-the-box speed-up solution.
Used as an analogy for how speculative decoding works, akin to auto-correction.
Used as an example of an edge device with limited memory bandwidth, where certain optimizations like Turbo Quantum Surf are highly effective.
A high-end NVIDIA GPU with high memory bandwidth, contrasting with edge devices in terms of optimization needs.
NVIDIA's AI supercomputing systems, mentioned in the context of heterogeneous topologies and interconnects.
A previous generation NVIDIA GPU, mentioned for its memory limitations compared to newer models like B200s.
An older NVIDIA GPU, referenced to highlight the progression of hardware capabilities.
An NVIDIA GPU architecture generation that preceded Hopper and Blackwell.
An NVIDIA GPU architecture generation, updated from Ampere and preceding Blackwell.
The latest NVIDIA GPU architecture, emphasizing NVFP4 and improved memory bandwidth.
NVIDIA's Grace Blackwell Superchip, mentioned for its memory capacity suitable for very large models like the 3-trillion parameter Kimmy.
More from Latent Space
View all 241 summaries
71 minOpenAI’s Plan to Make ChatGPT the Everything App — Akshay Nathan, OpenAI
117 minPoolside’s Model Factory, Laguna S, Open Models, and the Race to AGI — Eiso Kant, Poolside AI
90 minCausal Models Need Causal Data - Xaira’s X-Cell model (Bo Wang & Ci Chu)
50 minThe AI Memory Problem: Why Long Context Isn’t Enough — Dan Biderman, Engram Co-founder & CEO
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free