Arto Liukkonen
I broke the production
#1about 6 minutes
A personal story of breaking production at scale
The speaker recounts causing a major production outage by running a backfill script that overwhelmed the Facebook API and halted data updates.
#2about 2 minutes
Judging intentions versus actions during incidents
We tend to judge others by their actions but ourselves by our intentions, so we should assume good intent from colleagues during incidents.
#3about 2 minutes
Why individual blame is a counterproductive response
When a production issue occurs, it's a system failure, not an individual's fault, as responsibility is shared across developers, reviewers, and processes.
#4about 3 minutes
How to build a psychologically safe blameless culture
Shifting to a blameless culture requires fostering trust, understanding intentions, practicing self-awareness, and owning mistakes without displacing frustration.
#5about 2 minutes
Using blameless postmortems for system-level learning
Blameless postmortems, originating from aviation and healthcare, focus on investigating root causes to strengthen systems rather than assigning individual blame.
#6about 3 minutes
The power of positive feedback in code reviews
Applying the five-to-one ratio of positive to negative interactions can improve team dynamics, especially by adding positive comments during code reviews.
#7about 2 minutes
Using pre-mortems to proactively prevent failures
Pre-mortems are a proactive exercise where teams imagine a project has already failed in order to identify potential risks and edge cases beforehand.
#8about 3 minutes
Incident resolution and key cultural takeaways
The incident took 20 hours to fully resolve but was a valuable learning experience that exposed system flaws and reinforced a healthy team culture.
#9about 2 minutes
Q&A on customer impact and worst production breaks
The speaker answers audience questions about customer reactions to the outage and shares a story about his worst production break involving a failed form.
Related jobs
Jobs that call for the skills explored in this talk.
Featured Partners
Related Videos
29:38Answering the Million Dollar Question: Why did I Break Production?
Luís Ventura
28:36What we Learned from Reading 100+ Kubernetes Post-Mortems
Noaa Barki
27:52Chaos in Containers - Unleashing Resilience
Maish Saidel-Keesing
29:55Empathy: The secret sauce of Resilience
Malin Litwinski
27:27The Software Bug All Stars - and what we can learn from them
Christian Seifert
43:00Shipping Quality Software In Hostile Environments
Luka Kladaric
42:41Engineering culture: Why ownership is the secret ingredient
Eric Rabinovich
21:39Unveiling the Dark Side: Navigating the Pitfalls of Digital Ambitions
Johannes Hansen
From learning to earning
Jobs that call for the skills explored in this talk.
DevOps Engineer – Kubernetes & Cloud (m/w/d)
epostbox epb GmbH
Berlin, Germany
Intermediate
Senior
DevOps
Kubernetes
Cloud (AWS/Google/Azure)