Noaa Barki

What we Learned from Reading 100+ Kubernetes Post-Mortems

What's the #1 cause of Kubernetes outages? After analyzing over 100 post-mortems, the answer is surprisingly simple—and completely preventable.

What we Learned from Reading 100+ Kubernetes Post-Mortems
#1about 6 minutes

Understanding the developer versus DevOps cultural divide

A story from a DevOps meetup illustrates the different goals and perspectives that create friction between developers and operations teams.

#2about 2 minutes

Bridge the gap with champions and failure stories

Delegate knowledge to developer champions and learn best practices by studying the post-mortem stories of other companies.

#3about 5 minutes

Common Kubernetes misconfigurations from real outages

Examples from Target and Zalando show how simple errors like incorrect CronJob concurrency policies or missing memory limits can cause major production failures.

#4about 3 minutes

How to introduce policy enforcement gradually

Avoid organizational friction by implementing new policies slowly, starting with a single pilot team to gain agreement and understanding before a wider rollout.

#5about 3 minutes

Categorizing the three types of Kubernetes failures

Kubernetes failures typically fall into three categories: simple syntax errors, gaps in knowledge of best practices, and misalignment with internal company policies.

#6about 2 minutes

Validating Kubernetes YAML for syntax and schema errors

Use tools like yq for YAML format validation and kubeconform for schema validation without requiring direct cluster access for developers.

#7about 4 minutes

The challenges of managing policies as code in Git

Managing policies in Git creates versioning nightmares and lacks features for permissions, dynamic adjustments, and providing clear remediation guidelines.

#8about 4 minutes

Using Datree for centralized policy management

Datree is an open-source tool that provides a centralized location for managing policies, which are then enforced locally and in CI for developers.

#9about 1 minute

The real meaning of shifting responsibility left

True shift-left culture is not just about tools but about delegating responsibility and empowering developers to own their configurations.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

See all videos
Kubernetes Security - Challenge and Opportunity42:45

Kubernetes Security - Challenge and Opportunity

Marc Nimmerrichter

Hacking Kubernetes: Live Demo Marathon46:36

Hacking Kubernetes: Live Demo Marathon

Andrew Martin

Local Development Techniques with Kubernetes40:00

Local Development Techniques with Kubernetes

Rob Richardson

Retooling and refactoring - an investment in people.50:23

Retooling and refactoring - an investment in people.

Andrew Holway

Chaos in Containers - Unleashing Resilience27:52

Chaos in Containers - Unleashing Resilience

Maish Saidel-Keesing

Mastering Kubernetes – Beginner Edition57:24

Mastering Kubernetes – Beginner Edition

Hannes Norbert Göring

I broke the production26:18

I broke the production

Arto Liukkonen

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!29:21

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!

Um e Habiba

From learning to earning

Jobs that call for the skills explored in this talk.

DevOps con Kubernetes

EMETEL
Municipality of Madrid, Spain

Remote
Intermediate
GIT
Bash
Linux
DevOps
+11