Patrick Koss

Unveiling the Magic: Scaling Large Language Models to Serve Millions

A single short prompt can exhaust your GPU resources. Learn how a custom proxy and clever rate-limiting can serve large language models to millions of users.

Unveiling the Magic: Scaling Large Language Models to Serve Millions
#1about 3 minutes

Understanding the benefits of self-hosting large language models

Self-hosting LLMs provides greater control over data privacy, compliance, cost, and vendor lock-in compared to using third-party services.

#2about 4 minutes

Architectural overview for a scalable LLM serving platform

A scalable LLM service requires key components for model acquisition, inference, storage, billing, security, and request routing.

#3about 7 minutes

Choosing an inference engine and model storage strategy

Using network file storage (NFS) is crucial for reducing startup times and enabling fast horizontal scaling when deploying new model instances.

#4about 5 minutes

Building an efficient token-based billing system

Aggregate token usage with tools like Redis before sending data to a payment provider to manage rate limits and improve system efficiency.

#5about 3 minutes

Implementing robust rate limiting for shared LLM systems

Prevent system abuse by implementing both request-based and token-based rate limiting, using estimations for output tokens to protect shared resources.

#6about 3 minutes

Selecting the right authentication and authorization strategy

Bearer tokens offer a flexible solution for managing authentication and fine-grained authorization, such as restricting access to specific models.

#7about 2 minutes

Scaling inference with Kubernetes and smart routing

Use tools like KServe or Knative on Kubernetes for intelligent autoscaling and canary deployments based on custom metrics like queue size.

#8about 3 minutes

Summary of best practices for scalable LLM deployment

Key strategies for success include robust rate limiting, modular design, continuous benchmarking, and using canary deployments for safe production testing.

Related jobs
Jobs that call for the skills explored in this talk.

Machine Learning Engineer

Picnic Technologies B.V.

Picnic Technologies B.V.
Amsterdam, Netherlands

Intermediate
Senior
Python
Structured Query Language (SQL)
+1

Matching moments

Increasing the value of talk recordings post-event
04:57 MIN

Increasing the value of talk recordings post-event

Cat Herding with Lions and Tigers - Christian Heilmann

Organizing a developer conference for 15,000 attendees
01:32 MIN

Organizing a developer conference for 15,000 attendees

Cat Herding with Lions and Tigers - Christian Heilmann

Establishing a single source of truth for all data
02:39 MIN

Establishing a single source of truth for all data

Cat Herding with Lions and Tigers - Christian Heilmann

Automating video post-production with local scripts
02:54 MIN

Automating video post-production with local scripts

Cat Herding with Lions and Tigers - Christian Heilmann

Why corporate AI adoption lags behind the hype
03:28 MIN

Why corporate AI adoption lags behind the hype

What 2025 Taught Us: A Year-End Special with Hung Lee

Moving beyond headcount to solve business problems
04:27 MIN

Moving beyond headcount to solve business problems

What 2025 Taught Us: A Year-End Special with Hung Lee

Breaking down silos between HR, tech, and business
03:39 MIN

Breaking down silos between HR, tech, and business

What 2025 Taught Us: A Year-End Special with Hung Lee

Incentivizing automation with a 'keep what you kill' policy
05:18 MIN

Incentivizing automation with a 'keep what you kill' policy

What 2025 Taught Us: A Year-End Special with Hung Lee

Featured Partners

Related Videos

See all videos
Self-Hosted LLMs: From Zero to Inference30:04

Self-Hosted LLMs: From Zero to Inference

Roberto Carratalá & Cedric Clyburn

How AI Models Get Smarter35:16

How AI Models Get Smarter

Ankit Patel

Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

Using LLMs in your Product31:12

Using LLMs in your Product

Daniel Töws

Unlocking the Power of AI: Accessible Language Model Tuning for All31:50

Unlocking the Power of AI: Accessible Language Model Tuning for All

Cedric Clyburn & Legare Kerrison

How to Avoid LLM Pitfalls - Mete Atamel and Guillaume Laforge42:26

How to Avoid LLM Pitfalls - Mete Atamel and Guillaume Laforge

Meta Atamel & Guillaume Laforge

Inside the Mind of an LLM27:11

Inside the Mind of an LLM

Emanuele Fabbiani

Your Next AI Needs 10,000 GPUs. Now What?29:52

Your Next AI Needs 10,000 GPUs. Now What?

Anshul Jindal & Martin Piercy

Related Articles

View all articles
LM
Luis Minvielle
What Are Large Language Models?
Developers and writers can finally agree on one thing: Large Language Models, the subset of AIs that drive ChatGPT and its competitors, are stunning tech creations. Developers enjoying the likes of GitHub Copilot know the feeling: this new kind of te...
What Are Large Language Models?
KD
Krissy Davis
The Best Large Language Models on The Market
Large language models are sophisticated programs that enable machines to comprehend and generate human-like text. They have been the foundation of natural language processing for almost a decade. Although generative AI has only recently gained popula...
The Best Large Language Models on The Market
CH
Chris Heilmann
All the videos of Halfstack London 2024!
Last month was Halfstack London, a conference about the web, JavaScript and half a dozen other things. We were there to deliver a talk, but also to record all the sessions and we're happy to share them with you. It took a bit as we had to wait for th...
All the videos of Halfstack London 2024!

From learning to earning

Jobs that call for the skills explored in this talk.