Anatomy of an AI Request: Where Latency and Cost Are Really Born

Every LLM API call looks simple on the surface. But under the hood, it’s a deeply layered systems problem spanning kernels, compilers, GPU scheduling, and distributed inference infrastructure. In this session, Dan Fu, VP of Kernels at Together AI, breaks down what actually happens when a request hits a modern AI model—and how inference performance is ultimately determined. He walks through the full inference stack, from tokenization and model execution to GPU kernel dispatch, memory movement, and serving-time orchestration, highlighting where inefficiencies accumulate and why today’s systems operate far below theoretical hardware capability. Drawing on both cutting-edge systems research and production-scale infrastructure experience—including foundational work like FlashAttention, now widely adopted across the AI ecosystem—Dan unpacks the core bottlenecks in modern inference stacks. He also outlines where the largest efficiency gains are still available, and why kernel-level optimization is becoming a critical lever in scaling AI systems.