Unveiling the Magic: Scaling Large Language Models to Serve Millions

Ever wondered what’s behind the curtain when you chat with AI like OpenAI’s ChatGPT?

At STACKIT, we’ve developed a model-serving service designed to host chat models, embedding models, and more, with a particular focus on supporting large language models.

In this talk, we’ll unveil the challenges and solutions of scaling large language models (LLMs) to serve millions. We’ll explore model acquisition, caching, building an OpenAI-compatible server, secure authentication, usage tracking for billing, autoscaling, and hosting multiple models under the domain.

Whether you’re a developer, engineer, or just curious, this talk will show you how every token counts in making AI accessible to millions.