A Hands-On Developer Guide to Inference Engineering

Inference engineers solve a multi-dimensional puzzle across latency, throughput, and cost to serve generative AI models. Spanning interdependent layers of the serving stack, from CUDA to runtimes to containers to Kubernetes, inference engineering is the discipline behind scaling AI applications. In this session, we’ll cover key inference engineering concepts across both runtime and infrastructure, including topology-aware model parallelism, prefill-decode disaggregated serving, KV-aware routing, autoscaling strategies, and multi-cluster infrastructure management, with a focus on the work required to run multi-trillion-parameter LLMs efficiently in production.