How 8 GPUs Became 16: Rethinking LLM Inference Through Parallelism, Memory, and Runtime Design

Most teams scale LLM inference by adding more GPUs. This talk shows why that approach is fundamentally flawed—and how rethinking parallelism strategy, memory layout, and runtime configuration can deliver up to 2× production throughput improvements without changing hardware.

I will break down a real-world high-concurrency inference system serving millions of users, where the key breakthrough came not from bigger clusters but from architectural decisions: tensor vs data parallelism trade-offs, KV cache optimization, and Mixture-of-Experts routing efficiency.

The session dives deep into vLLM internals, FP8 quantization impacts, KV cache memory tuning, and GPU communication overheads, showing how reducing synchronization and improving locality dramatically changes system performance.