Patrick Koss

Unveiling the Magic: Scaling Large Language Models to Serve Millions

A single short prompt can exhaust your GPU resources. Learn how a custom proxy and clever rate-limiting can serve large language models to millions of users.

Unveiling the Magic: Scaling Large Language Models to Serve Millions
#1about 3 minutes

Understanding the benefits of self-hosting large language models

Self-hosting LLMs provides greater control over data privacy, compliance, cost, and vendor lock-in compared to using third-party services.

#2about 4 minutes

Architectural overview for a scalable LLM serving platform

A scalable LLM service requires key components for model acquisition, inference, storage, billing, security, and request routing.

#3about 7 minutes

Choosing an inference engine and model storage strategy

Using network file storage (NFS) is crucial for reducing startup times and enabling fast horizontal scaling when deploying new model instances.

#4about 5 minutes

Building an efficient token-based billing system

Aggregate token usage with tools like Redis before sending data to a payment provider to manage rate limits and improve system efficiency.

#5about 3 minutes

Implementing robust rate limiting for shared LLM systems

Prevent system abuse by implementing both request-based and token-based rate limiting, using estimations for output tokens to protect shared resources.

#6about 3 minutes

Selecting the right authentication and authorization strategy

Bearer tokens offer a flexible solution for managing authentication and fine-grained authorization, such as restricting access to specific models.

#7about 2 minutes

Scaling inference with Kubernetes and smart routing

Use tools like KServe or Knative on Kubernetes for intelligent autoscaling and canary deployments based on custom metrics like queue size.

#8about 3 minutes

Summary of best practices for scalable LLM deployment

Key strategies for success include robust rate limiting, modular design, continuous benchmarking, and using canary deployments for safe production testing.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

Self-Hosted LLMs: From Zero to Inference30:04

Self-Hosted LLMs: From Zero to Inference

Roberto Carratalá, Cedric Clyburn

How AI Models Get Smarter35:16

How AI Models Get Smarter

Ankit Patel

Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

Using LLMs in your Product31:12

Using LLMs in your Product

Daniel Töws

Inside the Mind of an LLM27:11

Inside the Mind of an LLM

Emanuele Fabbiani

How to Avoid LLM Pitfalls - Mete Atamel and Guillaume Laforge42:26

How to Avoid LLM Pitfalls - Mete Atamel and Guillaume Laforge

Meta Atamel, Guillaume Laforge

Your Next AI Needs 10,000 GPUs. Now What?29:52

Your Next AI Needs 10,000 GPUs. Now What?

Anshul Jindal, Martin Piercy

Exploring LLMs across clouds28:38

Exploring LLMs across clouds

Tomislav Tipurić

From learning to earning

Jobs that call for the skills explored in this talk.

AI/ML Engineer

Licorne Society
Canton of Toulouse-5, France

C++
GIT
CMake
Python
PyTorch
+2