WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated

Christin Pohl

Maximizing open-source LLM throughput requires aggressively balancing compute and memory bottlenecks. Master the complete GPU optimization stack, from simple model quantization to sophisticated speculative decoding.

Pause
Mute Enter Fullscreen
#1 about 2 min

Strategic reasons to adopt open-source inference models

Overcome vendor lock-in and high scaling costs by utilizing independent open-source models for customized agentic scenarios.

#2 about 2 min

Navigating the layers of the language model inference stack

Make open-source models operate efficiently by tuning the underlying infrastructure, framework layers, and dynamic runtime caching systems.

#3 about 3 min

Balancing compute and memory bound inference performance metrics

Balance competing performance indicators by separating compute-heavy initial token generation metrics from memory-bound subsequent latency measurements.

#4 about 3 min

Right-sizing language models for specific agentic use cases

Prevent unnecessary structural bloat by employing leaderboard tracking and custom evaluations to pinpoint correctly scaled language models.

#5 about 2 min

Leveraging quantization to improve memory utilization and throughput

Shrink memory footprints and escalate throughput simultaneously by applying reduced floating point precision to architectural components.

#6 about 2 min

Implementing smart routing and hardcoded intents for cost savings

Cut excessive inference overhead by triaging complex requests and hardcoding responses for simple recurring user greetings.

#7 about 2 min

Enhancing hardware utilization with continuous batching library frameworks

Eliminate idle hardware downtime by substituting static batch configurations with continuous frameworks that constantly process overlapping token sequences.

#8 about 2 min

Improving inference efficiency utilizing cache aware replica routing

Avoid recomputing identical conversational histories by routing recurring agentic prompts toward replicas already storing corresponding cache markers.

#9 about 3 min

Distributing large models effectively across multi-server architectures

Bypass severe hardware limitations by slicing neural network layers sequentially and parallelly across networked multi-gpu virtual machines.

#10 about 3 min

Accelerating token generation speeds with speculative decoding techniques

Resolve slow text production bottlenecks by rapidly projecting probable outputs using secondary draft networks for batch verification.

#11 about 2 min

Advancing efficiency with disaggregated inference and model distillation

Maximize high-volume stability by physically isolating request pre-validation nodes apart from memory-intensive continuous token decoding workers.

#12 about 3 min

Streamlining managed open-source deployments via native cloud resources

Abstract away complex hardware provisioning by integrating pre-optimized managed compute templates designed for broad open-source collections.

#13 about 2 min

Self-hosting distributed inference clusters on orchestrated container services

Achieve granular configuration access by managing custom open-source inference stacks directly over containerized graphic processing clusters.

#14 about 3 min

A structured framework for adopting custom open-source inference

Ensure predictable self-hosted deployments by advancing from defined benchmarks through structured machine learning operation scaling loops.

#15 about 2 min

Optimizing small models for local inference and personal usage

Attain absolute parameter control alongside zero runtime costs by adapting local inference structures toward small-scale personal devices.

Matching moments

2:39 min

Open source tools for running and scaling models

Cedric Clyburn Cedric Clyburn +1 · WWC 2025

3:30 min

Optimizing performance using dedicated open source inference engines

Patrick Koss Patrick Koss · WWC 2025

1:47 min

Managing compute costs and AI model routing

Deivids Vilkinsons Deivids Vilkinsons +3 · WWC Europe 2026

1:52 min

Prioritizing developer user experience over raw model parameters

Neel Sundaresan Neel Sundaresan +1 · WWC Europe 2026

1:45 min

Introduction to serving large language models locally

Patrick Koss Patrick Koss · WWC 2025

3:11 min

Optimizing inference costs through efficient distributed runtimes

Anshul Jindal Anshul Jindal +1 · WWC Europe 2026

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 1 month ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 1 month ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 1 month ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 1 month ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 1 month ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 1 month ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Fast, Cheap, and Accurate: Optimizing LLM Inference with vLLM and Quantization

Legare Kerrison, Cedric Clyburn

Legare Kerrison
Cedric Clyburn
Open session

World Congress 2026 North America

Agents That Own Their Inference: Building Production AI Agents on Dedicated GPUs

Duan Lightfoot

Sr. AI Engineer, Akamai

Duan Lightfoot
Open session

World Congress 2026 North America

Understanding LLM Architectures: Inside the Design of Modern Models

Jofia Jose Prakash

Enterprise AI Architect at American Chemical Society

Jofia Jose Prakash
Open session

World Congress 2026 North America

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong
Open session

World Congress 2026 North America

Headroom: A Context Optimization Layer for LLM Applications

Tejas Chopra

Senior Software Engineer at Netflix

Tejas Chopra
Open session

World Congress 2026 North America

Trust, But Verify: Continuous GPU Validation at Scale

Kyle Bell

VP of AI @ TensorWave

Kyle Bell

Related videos

Efficient deployment and inference of GPU-accelerated LLMs​ thumbnail
23:01
WWC 2024 • almost 2 years ago

Efficient deployment and inference of GPU-accelerated LLMs​

Adolf Hohl

Unveiling the Magic: Scaling Large Language Models to Serve Millions thumbnail
31:25
WWC 2025 • 12 months ago

Unveiling the Magic: Scaling Large Language Models to Serve Millions

Patrick Koss

Self-Hosted LLMs: From Zero to Inference thumbnail
30:04
WWC 2025 • 12 months ago

Self-Hosted LLMs: From Zero to Inference

Cedric Clyburn, Roberto Carratalá

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source thumbnail
29:50
WWC Europe 2026 • about 1 month ago

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source

Viktoria Semaan

Unlocking the Power of AI: Accessible Language Model Tuning for All thumbnail
31:50
WWC 2024 • almost 2 years ago

Unlocking the Power of AI: Accessible Language Model Tuning for All

Cedric Clyburn, @technicallylegare

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow thumbnail
34:21
LIVE • almost 2 years ago

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow

Aarno Aukia

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.