> Markdown version of [/videos/495-what-we-learned-from-reading-100-kubernetes-post-mortems](https://www.wearedevelopers.com/videos/495-what-we-learned-from-reading-100-kubernetes-post-mortems). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# What we Learned from Reading 100+ Kubernetes Post-Mortems

Subtle YAML errors and missing memory limits are silently crashing your Kubernetes clusters. Uncover the leading root causes from 100+ post-mortems and automate infrastructure validation directly within CI workflows.

- **Speakers:** Noaa Barki
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 28:36
- **URL:** https://www.wearedevelopers.com/videos/495-what-we-learned-from-reading-100-kubernetes-post-mortems

## Summary

The gap between DevOps engineers and developers often leads to Kubernetes cluster failures and production downtime. By analyzing over 100 Kubernetes post-mortems, the root causes of misconfigurations typically trace back to missing memory limits, runaway concurrency bounds, subtle YAML syntax errors, and general knowledge gaps among diverse personas. Bridging this divide requires shifting responsibility left and empowering developers through targeted education, such as picking full-stack 'champions' to propagate best practices incrementally rather than dropping massive policy restrictions all at once.

Cultivating a robust DevOps culture means validating resources automatically at every code change, treating infrastructure configurations with the exact same rigor as application source code. Organizations should implement local and CI-based validation steps using open-source tools like the `yq` processor for file formatting and `kubeconform` to verify structural schemas without requiring direct Kubernetes API or cluster access. Providing developers with local testing libraries aligns infrastructure code with the workflows they already practice daily, dramatically reducing friction.

Relying solely on version control to manage complex cluster rules falls short in scaling developer autonomy. Dynamic, centralized policy management solutions like Datree ensure consistent organizational standards across fragmented microservices while distributing rules seamlessly through GitOps pipelines. Crucially, when automated policies block a CI pipeline, providing transparent, detailed guidelines on why the block occurred and exactly how to fix the misconfiguration prevents developer frustration and permanently stops repeat errors.

**Keywords:** kubernetes misconfigurations, cluster stability, infrastructure as code management, shift left responsibility, kubernetes post-mortems, cross-functional devops culture, automated policy enforcement, kubernetes resource validation, CI pipeline automation, centralized policy management, gitops lifecycle, local testing libraries, kubeconform schema validation, datree integration, yq YAML processing, container memory limits, infrastructure code validation

## Chapters

1. **Friction between developers and infrastructure teams** (00:05) — Developers prioritize feature delivery over infrastructure concepts like memory limits and resource configurations.
1. **Finding developers to champion infrastructure best practices** (06:45) — Identifying team members interested in both code and infrastructure helps bridge the knowledge gap.
1. **Common Kubernetes misconfigurations in production environments** (08:39) — Real-world examples show how missing concurrency policies or memory limits cause major performance outages.
1. **Rolling out policy enforcement to engineering teams slowly** (12:36) — Gradually introducing automated validations in local and CI environments prevents friction while protecting production.
1. **Three primary categories of Kubernetes infrastructure failures** (15:48) — Most production outages stem from YAML syntax errors, misunderstanding tool defaults, or misaligned internal practices.
1. **Validating resource syntax and schema without cluster access** (18:33) — Tools like yq and kubeconform ensure basic file format and schema correctness before deployment.
1. **Managing infrastructure policies and guiding developers effectively** (20:40) — Centralizing policy management and providing detailed remediation steps prevents developer frustration when builds fail.
1. **Automating customized policy enforcement with open source tools** (23:47) — Implementing tools like Datree or Conftest allows centralized policy control while enabling local developer testing.

## Related Moments

- [Assessing common Kubernetes security incidents and misconfigurations](https://www.wearedevelopers.com/videos/1411-kubernetes-security-best-practices) (from "Kubernetes Security Best Practices")
- [Automating infrastructure to eliminate traditional DevOps roles](https://www.wearedevelopers.com/videos/371-retooling-and-refactoring-an-investment-in-people) (from "Retooling and refactoring - an investment in people.")
- [Introduction to Kubernetes security challenges and opportunities](https://www.wearedevelopers.com/videos/412-kubernetes-security-challenge-and-opportunity) (from "Kubernetes Security - Challenge and Opportunity")
- [Managing the complexity of bare metal Kubernetes deployments](https://www.wearedevelopers.com/videos/100135-the-new-shiny-syndrome-how-to-avoid-tech-hype-traps) (from "The New Shiny Syndrome: How to Avoid Tech Hype Traps")
- [Managing Kubernetes clusters using command line and YAML](https://www.wearedevelopers.com/videos/530-mastering-kubernetes-beginner-edition) (from "Mastering Kubernetes – Beginner Edition")
- [Managing deployments with Kubernetes and Terraform self-healing](https://www.wearedevelopers.com/videos/531-implementing-feature-environments-with-aws-and-terraform) (from "Implementing Feature Environments with AWS and Terraform")

## Related Articles

- [Learning Kubernetes made easy with KubeCampus](https://www.wearedevelopers.com/magazine/348-learning-kubernetes-made-easy-with-kubecampus)
- [Exploring AI: Opportunities and Risks for Developers](https://www.wearedevelopers.com/magazine/522-exploring-ai-opportunities-and-risks-for-developers)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Dev Digest 150 - The shift to AI generated code, fingerprinting and OKRs vs. doing your job](https://www.wearedevelopers.com/magazine/533-dev-digest-150-the-shift-to-ai-generated-code-fingerprinting-and-okrs-vs-doing-your-job)

## Related Jobs

- [Platform Engineer (DevOps)](https://www.wearedevelopers.com/jobs/48264-platform-engineer-devops) at **WDW Consulting GmbH**
- [Lead Cloud DevSecOps Engineer - Kubernetes](https://www.wearedevelopers.com/jobs/ext/1659167-lead-cloud-devsecops-engineer-kubernetes) at **BWI GmbH**
- [Devops Engineer](https://www.wearedevelopers.com/jobs/ext/1940926-devops-engineer) at **Bitpanda**
- [Senior Software Engineer, Enterprise Products](https://www.wearedevelopers.com/jobs/ext/1841248-senior-software-engineer-enterprise-products) at **GitHub**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**