> Markdown version of [/videos/1130-chatbots-are-going-to-destroy-infrastructures-and-your-cloud-bills?t=353](https://www.wearedevelopers.com/videos/1130-chatbots-are-going-to-destroy-infrastructures-and-your-cloud-bills?t=353). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Chatbots are going to destroy infrastructures and your cloud bills

Is your new RAG chatbot secretly bankrupting your cloud budget? Monolithic architectures bottleneck servers and skyrocket costs. Learn how decoupling heavy GPU tasks saves your infrastructure.

- **Speakers:** [Stan Girard](https://www.wearedevelopers.com/@stan-girard)
- **Event:** World Congress 2024
- **Published:** August 20, 2024
- **Duration:** 21:21
- **URL:** https://www.wearedevelopers.com/videos/1130-chatbots-are-going-to-destroy-infrastructures-and-your-cloud-bills

## Summary

The rise of generative AI has created a new breed of AI engineers, forcing a collision between lightweight web development and resource-heavy data science. As developers rush to build Retrieval-Augmented Generation (RAG) applications, they often underestimate the architectural shift required. Evolving a simple prototype into a production-ready chatbot capable of document ingestion and OCR transforms lightweight, CPU-bound code into massive, GPU-reliant bottlenecks. This fatal architecture flaw results in bloated Docker images, unbearable user latency, and unsustainable operational costs. As the speaker warns, chatbots are going to destroy infrastructures and your cloud bills if fundamental principles are ignored.

To prevent cost overruns, engineering teams must abandon monolithic AI architectures. The crucial pivot involves splitting codebases into a service-oriented architecture, strictly decoupling fast, synchronous web services—like UI and user authentication via Supabase or PostgreSQL—from slow, asynchronous GPU-bound background tasks, such as embeddings, vector database retrieval, and LangChain orchestration. Avoid the temptation to self-host LLMs or mandate on-premise deployments too early; instead, leverage external APIs to bypass the nightmare of provisioning GPUs on Kubernetes, which is notoriously difficult to maintain and scale efficiently.

Real-world optimization goes beyond simply separating architectural concerns. Latency is a primary killer of RAG usability, shifting standard 150-millisecond web responses to grueling three-to-four-second waits due to sequential retrieval and generation steps. Furthermore, engineering teams must evaluate model sizing rigorously. Through a novel hackathon experiment forcing AI models to play Street Fighter in real-time, it was proven that smaller models often outperform larger ones for specific tasks due to significantly lower latency and faster execution. Ultimately, quality doesn't always scale with the complexity of hosting larger models, making smaller, specialized, and API-driven tools the most pragmatic choice for early-stage infrastructure.

**Keywords:** rag infrastructure scaling, generative ai cloud bills, chatbot latency management, decoupling cpu and gpu workloads, docker container bloat, llm self-hosting challenges, service-oriented ai architectures, vector database integration, langchain orchestration bottlenecks, kubernetes gpu provisioning, retrieval-augmented generation performance, asynchronous background ai workers, deploying small vs large llms, open-source chatbot infrastructure

## Chapters

1. **Introduction to chatbot infrastructure and cloud challenges** (00:00) — How the rapid growth of an open-source chatbot project highlights the importance of avoiding structural infrastructure mistakes.
1. **Contrasting web developers and data scientists** (01:00) — Differences in resource constraints, scaling models, and deployment environments separate traditional software and data engineering roles.
1. **The new paradigm of AI engineers** (02:38) — AI engineering merges practices from both web and data science, introducing unique optimization and computing constraints.
1. **Understanding basic retrieval-augmented generation architectures in chatbots** (04:09) — A breakdown of RAG workflows entails document chunking, GPU-bound embeddings, and sequential model generation calls.
1. **Scaling bottlenecks in generative AI applications** (05:53) — The challenge of managing parallel execution, slow response times, and sequential steps complicates user-facing AI deployments.
1. **Building a lean initial chatbot prototype application** (07:32) — Using a minimal tech stack creates an efficient and lightweight artificial intelligence container application.
1. **Bloat and complexity from adding GPU-bound tasks** (09:11) — Introducing OCR and advanced embeddings drastically increases container image sizes and subsequent infrastructure scaling costs.
1. **Refactoring application architectures for complex technical tasks** (10:25) — Separating fast user-facing business logic from slow background workers handles complex orchestrations and integrations effectively.
1. **Infrastructure challenges of on-premise chatbot deployments** (12:18) — Hosting monolithic, multi-model AI applications on local organizational servers poses massive server tuning and scaling challenges.
1. **Architecting generative AI systems by targeted user load** (14:34) — Guidelines for splitting application components apply directly to managing continuous or spiky application traffic patterns.
1. **Best practices for utilizing large language models in production** (15:28) — Strategies favoring managed APIs and service-oriented architectures reduce the immense overhead of self-hosting models.
1. **Overlooked AI infrastructure and operational deployment barriers** (17:44) — Managing complex deployments like configuring GPUs on Kubernetes requires dedicated pipelines for backups and non-deterministic testing.
1. **Evaluating large language models using real-time video games** (19:14) — A hackathon approach to measuring model capability proves that smaller setups often outperform larger architectures in speed-sensitive tasks.

## Related Moments

- [Leveraging the comprehensive generative artificial intelligence stack](https://www.wearedevelopers.com/videos/969-make-it-simple-using-generative-ai-to-accelerate-learning) (from "Make it simple, using generative AI to accelerate learning")
- [Architectural patterns for developing robust generative AI applications](https://www.wearedevelopers.com/videos/1512-building-ai-applications-with-langchain-and-node-js) (from "Building AI Applications with LangChain and Node.js")
- [Progressing gracefully from generic chatbots to agentic workflows](https://www.wearedevelopers.com/videos/100148-ai-that-fits-your-business-not-the-other-way-around) (from "AI That Fits Your Business, Not the Other Way Around")
- [Constructing scalable AI solutions using LangChain and LangGraph](https://www.wearedevelopers.com/videos/1512-building-ai-applications-with-langchain-and-node-js) (from "Building AI Applications with LangChain and Node.js")
- [Evolution and challenges of building AI applications](https://www.wearedevelopers.com/videos/1601-one-ai-api-to-power-them-all) (from "One AI API to Power Them All")
- [Architectural components of a generative AI agent](https://www.wearedevelopers.com/videos/782-building-real-time-ai-ml-agents-with-distributed-data-using-apache-cassandra-and-astra-db) (from "Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB")

## Related Articles

- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)

## Related Jobs

- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio**