WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Unveiling the Magic: Scaling Large Language Models to Serve Millions

Patrick Koss

Ditch the 20-minute cold starts. Treat self-hosted LLMs as compute-heavy REST APIs. Learn how NFS, Pingora, and intelligent rate-limiting seamlessly scale your infrastructure to serve millions.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to serving large language models locally

Scaling large language models involves standard software engineering tricks layered on top of core AI systems.

#2 about 3 min

Compelling business reasons for self-hosting language models

Self-hosting ensures data privacy, compliance, and cost efficiency while avoiding vendor lock-in.

#3 about 4 min

Approaching language models as scalable synchronous rest APIs

Horizontal scaling of large language models requires caching, concurrency management, and distributed worker nodes.

#4 about 2 min

Sourcing and downloading model weights for fast access

Efficiently retrieving massive files from object storage or content delivery networks minimizes model boot times.

#5 about 4 min

Optimizing performance using dedicated open source inference engines

Inference engines manage memory caching and asynchronous worker thread pools to handle expensive computing tasks efficiently.

#6 about 2 min

Strategies for attaching scalable storage to virtual machines

Connecting instances to network file storage dramatically reduces deployment cold start times when spinning up additional capacity.

#7 about 5 min

Implementing efficient token aggregation and billing middleware proxies

Deploying a lightweight, high-performance proxy effectively tracks token usage for deferred customer billing.

#8 about 4 min

Preventing system exhaustion through token validation and estimations

Estimating output tokens prevents a single user's long request from monopolizing the shared computing infrastructure.

#9 about 4 min

Managing robust access credentials and API gateway limits

Utilizing mutual TLS or API keys controls secure access to specific inference endpoints.

#10 about 2 min

Deploying models dynamically with distributed inference load balancers

Intelligent request routing paired with dynamic scaling prevents unpredictable traffic spikes to overloaded worker nodes.

#11 about 4 min

Best practices for architecture stability and production testing

Enforcing system constraints and executing lightweight canary deployments reduces operational costs and infrastructure risk.

Matching moments

2:16 min

Best practices for utilizing large language models in production

Stan Girard Stan Girard · WWC 2024

2:39 min

Open source tools for running and scaling models

Cedric Clyburn Cedric Clyburn +1 · WWC 2025

3:14 min

Balancing performance and costs with custom language models

Jemiah Sius Jemiah Sius · WWC 2024

4:59 min

Optimizing server architecture for scalable large language model operations

Calvin Seward Calvin Seward · Europe 2026 Virtual

5:25 min

Addressing core challenges in large language model deployments

Vijay Krishan Gupta +1 · LIVE

1:20 min

Reducing infrastructure capacity requirements by deploying small language models

Alexander Wallner Alexander Wallner +3 · WWC 2024

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Fast, Cheap, and Accurate: Optimizing LLM Inference with vLLM and Quantization

Cedric Clyburn, Legare Kerrison

Cedric Clyburn
Legare Kerrison
Open session

World Congress 2026 North America

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong
Open session

World Congress 2026 North America

Understanding LLM Architectures: Inside the Design of Modern Models

Jofia Jose Prakash

Enterprise AI Architect at American Chemical Society

Jofia Jose Prakash
Open session

World Congress 2026 North America

Agents That Own Their Inference: Building Production AI Agents on Dedicated GPUs

Duan Lightfoot

Sr. AI Engineer, Akamai

Duan Lightfoot
Open session

World Congress 2026 North America

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

Michal Salanci

Senior Systems Engineer at ESET Cybersecurity

Michal Salanci
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan

Related videos

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated thumbnail
30:40
WWC Europe 2026 • about 1 month ago

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated

Christin Pohl

Self-Hosted LLMs: From Zero to Inference thumbnail
30:04
WWC 2025 • 12 months ago

Self-Hosted LLMs: From Zero to Inference

Cedric Clyburn, Roberto Carratalá

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow thumbnail
34:21
LIVE • almost 2 years ago

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow

Aarno Aukia

Efficient deployment and inference of GPU-accelerated LLMs​ thumbnail
23:01
WWC 2024 • almost 2 years ago

Efficient deployment and inference of GPU-accelerated LLMs​

Adolf Hohl

Unlocking the Power of AI: Accessible Language Model Tuning for All thumbnail
31:50
WWC 2024 • almost 2 years ago

Unlocking the Power of AI: Accessible Language Model Tuning for All

Cedric Clyburn, @technicallylegare

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source thumbnail
29:50
WWC Europe 2026 • about 1 month ago

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source

Viktoria Semaan

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.