> Markdown version of [/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale?t=2255](https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale?t=2255). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# 5 steps for running a Kubernetes environment at scale

Hitting memory limits will instantly terminate your production pods. Master a five-step observability strategy to troubleshoot crash loops, monitor telemetry, and confidently run Kubernetes at scale.

- **Speakers:** Stijn Polfliet
- **Event:** WeAreDevelopers LIVE
- **Published:** December 3, 2020
- **Duration:** 57:09
- **URL:** https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale

## Summary

Scaling Kubernetes for production requires moving beyond basic deployment to a comprehensive strategy for observability. Left unmonitored, the dynamic scaling and auto-restarting behaviors of container orchestration can lead to unpredictable application performance. This session introduces a five-layer observability model designed to help developers and operators manage Kubernetes at scale with confidence, bridging the gap between infrastructure health and application-level performance using the New Relic platform.

To ensure long-term stability and security, teams must enforce strict resource requests and limits; exceeding CPU allocations merely throttles execution, but hitting memory limits causes immediate pod termination. Further optimizations rely on utilizing minimal, purpose-built container images, such as Alpine Linux, which drastically reduce build times and minimize the surface area for security vulnerabilities. Defining strict liveness and readiness probes is also essential for directing traffic safely and accurately informing the cluster scheduler about the true state of individual pods.

By leveraging tools like Fluent Bit for lightweight log forwarding and Cube State Metrics for object health tracking, operators can stitch together a complete picture of cluster behavior. Distributed tracing maps the complex communication pathways between microservices, revealing latency bottlenecks deep within the stack. Finally, harnessing existing open-source telemetry—like integrating Prometheus data into Grafana or analyzing metrics natively via PromQL—allows engineering teams to centralize their monitoring, troubleshoot crash loops efficiently, and maintain complete service observability without managing their own long-term storage infrastructure.

**Keywords:** kubernetes observability model, cluster capacity planning, container resource limits, readiness and liveness probes, microservice distributed tracing, fluent bit log management, prometheus metrics integration, grafana dashboard visualization, alpine linux container security, cube state metrics, debugging crashloopbackoff, pod health monitoring, promql queries, new relic cluster explorer, telemetry data platform

## Chapters

1. **Introduction to the five-layer Kubernetes observability model** (00:22) — A high-level five-layer framework helps teams monitor and scale containerized environments reliably.
1. **Identifying failing pods visually in the cluster explorer** (05:49) — Visualizing node and pod health provides immediate context on pending or crash-looping workloads.
1. **Managing node capacity using resource requests and limits** (11:07) — Tracking resource consumption alongside configured request limits prevents node starvation and unexpected out-of-memory errors.
1. **Using smaller container images for performance and security** (18:30) — Switching to lightweight distribution bases like Alpine Linux improves build times and minimizes potential security vulnerabilities.
1. **Controlling pod routing with readiness and liveness probes** (21:03) — Implementing readiness and liveness probes allows the scheduler to control traffic routing and automated pod restarts accurately.
1. **Centralizing Kubernetes microservice log collection with Fluent Bit** (25:09) — Forwarding microservice log streams via lightweight log shippers centralizes log search and debugging.
1. **Tracing distributed communication pathways across Kubernetes microservices** (28:07) — Injecting trace identifiers into HTTP headers reveals latency bottlenecks across complex inter-service communication paths.
1. **Visualizing Prometheus open metrics using custom Grafana dashboards** (37:35) — Ingesting open-source metric endpoints into a centralized platform unifies query workflows alongside distributed traces and events.
1. **Deploying the Kubernetes observability stack using Helm charts** (48:15) — Installing specialized monitoring components via package managers ensures continuous visibility alongside dynamic application deployments.

## Related Moments

- [Exploring advanced observability stacks and distributed infrastructure challenges](https://www.wearedevelopers.com/videos/544-plan-ci-cd-on-the-enterprise-level) (from "Plan CI/CD on the Enterprise level!")
- [Orchestrating with Kubernetes against Docker and accessing slides](https://www.wearedevelopers.com/videos/487-debug-a-kubernetes-operator) (from "Debug a Kubernetes Operator")
- [Introduction to Kubernetes security challenges and opportunities](https://www.wearedevelopers.com/videos/412-kubernetes-security-challenge-and-opportunity) (from "Kubernetes Security - Challenge and Opportunity")
- [Transitioning from monolith architectures to microservices and Kubernetes](https://www.wearedevelopers.com/videos/108-get-ready-for-operations-by-pull-requests) (from "Get ready for operations by pull requests")
- [Visualizing Kubernetes for alternative and creative learning styles](https://www.wearedevelopers.com/videos/100085-understanding-kubernetes-in-a-visual-way) (from "Understanding Kubernetes in a visual way")
- [Evaluating auto-scalable infrastructure and capacity resource requirements](https://www.wearedevelopers.com/videos/1171-answering-the-million-dollar-question-why-did-i-break-production) (from "Answering the Million Dollar Question: Why did I Break Production?")

## Related Articles

- [Learning Kubernetes made easy with KubeCampus](https://www.wearedevelopers.com/magazine/348-learning-kubernetes-made-easy-with-kubecampus)
- [Effortlessly Scale Prometheus With The Telemetry Data Platform – And Keep your Grafana Dashboards, Too!](https://www.wearedevelopers.com/magazine/3-effortlessly-scale-prometheus-with-the-telemetry-data-platform-and-keep-your-grafana-dashboards-too)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Building AI Solutions with Rust and Docker](https://www.wearedevelopers.com/magazine/494-building-ai-solutions-with-rust-and-docker)

## Related Jobs

- [Platform Engineer (DevOps)](https://www.wearedevelopers.com/jobs/48264-platform-engineer-devops) at **WDW Consulting GmbH**
- [Lead Cloud DevSecOps Engineer - Kubernetes](https://www.wearedevelopers.com/jobs/ext/1659167-lead-cloud-devsecops-engineer-kubernetes) at **BWI GmbH**
- [Devops Engineer](https://www.wearedevelopers.com/jobs/ext/1940926-devops-engineer) at **Bitpanda**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**
- [Platform Engineer - Mercury Runtime Platform](https://www.wearedevelopers.com/jobs/ext/293235-platform-engineer-mercury-runtime-platform) at **Raiffeisen Bank International AG**
- [DevOps Engineer (m/f/d)](https://www.wearedevelopers.com/jobs/48303-devops-engineer-m-f-d) at **basebox GmbH**