WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 9, 2026 • Session details

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated

Christin Pohl

Maximizing open-source LLM throughput requires aggressively balancing compute and memory bottlenecks. Master the complete GPU optimization stack, from simple model quantization to sophisticated speculative decoding.

Pause
Mute Enter Fullscreen
#1 about 2 min

Strategic reasons to adopt open-source inference models

Overcome vendor lock-in and high scaling costs by utilizing independent open-source models for customized agentic scenarios.

#2 about 2 min

Navigating the layers of the language model inference stack

Make open-source models operate efficiently by tuning the underlying infrastructure, framework layers, and dynamic runtime caching systems.

#3 about 3 min

Balancing compute and memory bound inference performance metrics

Balance competing performance indicators by separating compute-heavy initial token generation metrics from memory-bound subsequent latency measurements.

#4 about 3 min

Right-sizing language models for specific agentic use cases

Prevent unnecessary structural bloat by employing leaderboard tracking and custom evaluations to pinpoint correctly scaled language models.

#5 about 2 min

Leveraging quantization to improve memory utilization and throughput

Shrink memory footprints and escalate throughput simultaneously by applying reduced floating point precision to architectural components.

#6 about 2 min

Implementing smart routing and hardcoded intents for cost savings

Cut excessive inference overhead by triaging complex requests and hardcoding responses for simple recurring user greetings.

#7 about 2 min

Enhancing hardware utilization with continuous batching library frameworks

Eliminate idle hardware downtime by substituting static batch configurations with continuous frameworks that constantly process overlapping token sequences.

#8 about 2 min

Improving inference efficiency utilizing cache aware replica routing

Avoid recomputing identical conversational histories by routing recurring agentic prompts toward replicas already storing corresponding cache markers.

#9 about 3 min

Distributing large models effectively across multi-server architectures

Bypass severe hardware limitations by slicing neural network layers sequentially and parallelly across networked multi-gpu virtual machines.

#10 about 3 min

Accelerating token generation speeds with speculative decoding techniques

Resolve slow text production bottlenecks by rapidly projecting probable outputs using secondary draft networks for batch verification.

#11 about 2 min

Advancing efficiency with disaggregated inference and model distillation

Maximize high-volume stability by physically isolating request pre-validation nodes apart from memory-intensive continuous token decoding workers.

#12 about 3 min

Streamlining managed open-source deployments via native cloud resources

Abstract away complex hardware provisioning by integrating pre-optimized managed compute templates designed for broad open-source collections.

#13 about 2 min

Self-hosting distributed inference clusters on orchestrated container services

Achieve granular configuration access by managing custom open-source inference stacks directly over containerized graphic processing clusters.

#14 about 3 min

A structured framework for adopting custom open-source inference

Ensure predictable self-hosted deployments by advancing from defined benchmarks through structured machine learning operation scaling loops.

#15 about 2 min

Optimizing small models for local inference and personal usage

Attain absolute parameter control alongside zero runtime costs by adapting local inference structures toward small-scale personal devices.

Matching moments

2:39 min

Open source tools for running and scaling models

Cedric Clyburn Cedric Clyburn +1 · World Congress 2025

3:30 min

Optimizing performance using dedicated open source inference engines

Patrick Koss Patrick Koss · World Congress 2025

1:47 min

Managing compute costs and AI model routing

Deivids Vilkinsons Deivids Vilkinsons +3 · World Congress 2026 Europe

1:52 min

Prioritizing developer user experience over raw model parameters

Neel Sundaresan Neel Sundaresan +1 · World Congress 2026 Europe

1:45 min

Introduction to serving large language models locally

Patrick Koss Patrick Koss · World Congress 2025

3:11 min

Optimizing inference costs through efficient distributed runtimes

Anshul Jindal Anshul Jindal +1 · World Congress 2026 Europe

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Related videos

Efficient deployment and inference of GPU-accelerated LLMs​ thumbnail
23:01
World Congress 2024 • about 2 years ago

Efficient deployment and inference of GPU-accelerated LLMs​

Adolf Hohl

Unveiling the Magic: Scaling Large Language Models to Serve Millions thumbnail
31:25
World Congress 2025 • about 1 year ago

Unveiling the Magic: Scaling Large Language Models to Serve Millions

Patrick Koss

Self-Hosted LLMs: From Zero to Inference thumbnail
30:04
World Congress 2025 • about 1 year ago

Self-Hosted LLMs: From Zero to Inference

Cedric Clyburn, Roberto Carratalá

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source thumbnail
29:50
World Congress 2026 Europe • 3 months ago

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source

Viktoria Semaan

Unlocking the Power of AI: Accessible Language Model Tuning for All thumbnail
31:50
World Congress 2024 • about 2 years ago

Unlocking the Power of AI: Accessible Language Model Tuning for All

Cedric Clyburn, @technicallylegare

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow thumbnail
34:21
LIVE • almost 2 years ago

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow

Aarno Aukia

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.