WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Unveiling the Magic: Scaling Large Language Models to Serve Millions

Patrick Koss

Ditch the 20-minute cold starts. Treat self-hosted LLMs as compute-heavy REST APIs. Learn how NFS, Pingora, and intelligent rate-limiting seamlessly scale your infrastructure to serve millions.

Pause
Mute Enter Fullscreen
#1 about 2 min

Introduction to serving large language models locally

Scaling large language models involves standard software engineering tricks layered on top of core AI systems.

#2 about 3 min

Compelling business reasons for self-hosting language models

Self-hosting ensures data privacy, compliance, and cost efficiency while avoiding vendor lock-in.

#3 about 4 min

Approaching language models as scalable synchronous rest APIs

Horizontal scaling of large language models requires caching, concurrency management, and distributed worker nodes.

#4 about 2 min

Sourcing and downloading model weights for fast access

Efficiently retrieving massive files from object storage or content delivery networks minimizes model boot times.

#5 about 4 min

Optimizing performance using dedicated open source inference engines

Inference engines manage memory caching and asynchronous worker thread pools to handle expensive computing tasks efficiently.

#6 about 2 min

Strategies for attaching scalable storage to virtual machines

Connecting instances to network file storage dramatically reduces deployment cold start times when spinning up additional capacity.

#7 about 5 min

Implementing efficient token aggregation and billing middleware proxies

Deploying a lightweight, high-performance proxy effectively tracks token usage for deferred customer billing.

#8 about 4 min

Preventing system exhaustion through token validation and estimations

Estimating output tokens prevents a single user's long request from monopolizing the shared computing infrastructure.

#9 about 4 min

Managing robust access credentials and API gateway limits

Utilizing mutual TLS or API keys controls secure access to specific inference endpoints.

#10 about 2 min

Deploying models dynamically with distributed inference load balancers

Intelligent request routing paired with dynamic scaling prevents unpredictable traffic spikes to overloaded worker nodes.

#11 about 4 min

Best practices for architecture stability and production testing

Enforcing system constraints and executing lightweight canary deployments reduces operational costs and infrastructure risk.

Matching moments

2:16 min

Best practices for utilizing large language models in production

Stan Girard Stan Girard · World Congress 2024

2:39 min

Open source tools for running and scaling models

Cedric Clyburn Cedric Clyburn +1 · World Congress 2025

3:14 min

Balancing performance and costs with custom language models

Jemiah Sius Jemiah Sius · World Congress 2024

4:59 min

Optimizing server architecture for scalable large language model operations

Calvin Seward Calvin Seward · Europe 2026 Virtual

5:25 min

Addressing core challenges in large language model deployments

Vijay Krishan Gupta +1 · LIVE

1:20 min

Reducing infrastructure capacity requirements by deploying small language models

Alexander Wallner Alexander Wallner +3 · World Congress 2024

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
11 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
about 1 year ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
about 1 year ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
about 1 year ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
about 1 year ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
about 1 year ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 16:10–16:40

Stage 9

Fast, Cheap, and Accurate: Optimizing LLM Inference with vLLM and Quantization

Legare Kerrison, Cedric Clyburn

Legare Kerrison
Cedric Clyburn
Open session

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 6

No Single Model to Rule Them All: Building Resilient AI Agents Across Open & Closed LLMs

Emmanuel Acheampong

Senior Manager Developer Relations at Crusoe AI

Emmanuel Acheampong
Open session

World Congress 2026 North America

September 24, 2026 · 13:30–14:00

Stage 9

Understanding LLM Architectures: Inside the Design of Modern Models

Jofia Jose Prakash

Director - AI & Governance at Humanity + AI, Inc

Jofia Jose Prakash
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 1

Anatomy of an AI Request: Where Latency and Cost Are Really Born

Dan Fu

VP of Kernels at Together AI

Dan Fu
Open session

World Congress 2026 North America

September 23, 2026 · 10:45–12:45

Stage 10

Agents That Own Their Inference: Building Production AI Agents on Dedicated GPUs

Duan Lightfoot

Sr. AI Engineer, Akamai

Duan Lightfoot
Open session

World Congress 2026 North America

September 24, 2026 · 15:30–16:00

Stage 9

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

Michal Salanci

Senior Systems Engineer at ESET Cybersecurity

Michal Salanci

Related videos

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated thumbnail
30:40
World Congress 2026 Europe • 2 months ago

Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated

Christin Pohl

Self-Hosted LLMs: From Zero to Inference thumbnail
30:04
World Congress 2025 • about 1 year ago

Self-Hosted LLMs: From Zero to Inference

Cedric Clyburn, Roberto Carratalá

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow thumbnail
34:21
LIVE • almost 2 years ago

DevOps for AI: running LLMs in production with Kubernetes and KubeFlow

Aarno Aukia

Efficient deployment and inference of GPU-accelerated LLMs​ thumbnail
23:01
World Congress 2024 • about 2 years ago

Efficient deployment and inference of GPU-accelerated LLMs​

Adolf Hohl

Unlocking the Power of AI: Accessible Language Model Tuning for All thumbnail
31:50
World Congress 2024 • about 2 years ago

Unlocking the Power of AI: Accessible Language Model Tuning for All

Cedric Clyburn, @technicallylegare

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source thumbnail
29:50
World Congress 2026 Europe • 2 months ago

Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source

Viktoria Semaan

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.