WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2022 • Jun 15, 2022

I broke the production

Arto Liukkonen

A new hire brought down production with a rogue script. Discover why resilient engineering teams embrace blameless post-mortems to fortify systems instead of hunting for a scapegoat.

Pause
Mute Enter Fullscreen
#1 about 3 min

Recognizing the inevitability of breaking production environments

Sharing early career failures normalizes the reality that all developers will eventually cause widespread system chaos.

#2 about 3 min

Managing constant production updates at massive data scale

Processing petabytes of advertising data requires tightly managed continuous deployment pipelines supporting dozens of daily updates.

#3 about 3 min

Triggering unexpected outages during restrictive data backfills

Attempting massive data backfills directly in production due to compliance restrictions can easily trigger severe monitoring spikes.

#4 about 3 min

Reconciling individual actions with good technical intentions

Overcoming the human tendency to judge coworkers by actions rather than intentions fosters fairer incident assessments.

#5 about 3 min

Recognizing systemic flaws over pointing fingers at developers

Treating deployment outages as failures of the pull request process rather than individual mistakes prevents toxic blame loops.

#6 about 2 min

Fostering self awareness to prevent toxic team behaviors

Cultivating personal responsibility and empathy stops frustration from cascading downward into harmful workplace chain reactions.

#7 about 2 min

Implementing blameless postmortems to strengthen system resilience

Borrowing practices from the healthcare and aviation industries turns every technical mistake into an opportunity for infrastructure improvement.

#8 about 4 min

Applying positive feedback ratios to technical code reviews

Leverging a five-to-one positive feedback model during pull requests builds team trust and breaks the cycle of purely critical interactions.

#9 about 2 min

Anticipating systemic failures using proactive premortem exercises

Simulating disaster scenarios before code ships helps uncover hidden edge cases and vulnerabilities within complex architectures.

#10 about 4 min

Embracing organizational learning following stressful production incidents

Navigating extended recovery windows proves that eliminating blame allows teams to focus entirely on rapid system restoration.

#11 about 3 min

Handling customer impact and mitigating third party failures

Buffering incidents through customer support provides necessary space for engineers while past external routing failures offer valuable historical lessons.

Matching moments

2:08 min

Navigating and mitigating the impacts of broken engineering cultures

Oskar Kruschitz Oskar Kruschitz · Europe 2026 Virtual

4:40 min

Managing continuous deployment through rigorous blameless postmortems

Markus Dorner · Coffee With Developers

1:49 min

The feedback loop of leadership choices and technical outcomes

Oskar Kruschitz Oskar Kruschitz · Europe 2026 Virtual

3:03 min

Building a corporate culture that actively celebrates failure

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

3:34 min

Writing blameless and detailed incident postmortems

Martin Beránek · LIVE

1:54 min

Embracing failures and analyzing high-profile software engineering mistakes

Christian Seifert · World Congress 2022

More from World Congress 2022

What the heck do "declarative" and "reactive" actually mean? thumbnail
30:40
about 4 years ago

What the heck do "declarative" and "reactive" actually mean?

André Kovac

The Design and Evolution of C++ thumbnail
1:01:15
over 4 years ago

The Design and Evolution of C++

Bjarne Stroustrup

But, you're not Facebook thumbnail
31:12
over 4 years ago

But, you're not Facebook

Kristijan Ristovski

Communicate efficiently with Software Architecture Diagrams thumbnail
28:11
over 4 years ago

Communicate efficiently with Software Architecture Diagrams

Simon Lasselsberger

Get Ready for Jakarta EE 10! thumbnail
42:53
over 4 years ago

Get Ready for Jakarta EE 10!

Ivar Grimstad

Keep your code refactorable thumbnail
26:46
over 4 years ago

Keep your code refactorable

Gerrit Stapper

Related videos

Blameless Postmortems That Change Nothing: Cultural Anti-Patterns in Incident Learning thumbnail
28:02
World Congress 2026 Europe • 3 months ago

Blameless Postmortems That Change Nothing: Cultural Anti-Patterns in Incident Learning

Daniel Schley

Empathy: The secret sauce of Resilience thumbnail
29:55
World Congress 2022 • over 4 years ago

Empathy: The secret sauce of Resilience

Malin Litwinski

Answering the Million Dollar Question: Why did I Break Production? thumbnail
29:38
World Congress 2024 • about 2 years ago

Answering the Million Dollar Question: Why did I Break Production?

Luís Ventura

Fail fast. How embracing failure leads to innovation at ZEISS. thumbnail
27:40
World Congress 2024 • about 2 years ago

Fail fast. How embracing failure leads to innovation at ZEISS.

Katharina Enßle

From Code to Culture: Why Leadership Determines Software Quality thumbnail
30:07
Europe 2026 Virtual • 3 months ago

From Code to Culture: Why Leadership Determines Software Quality

Oskar Kruschitz

How to Survive The Worst Launch of Your Career with Brenda Romero thumbnail
30:53
World Congress 2023 • about 3 years ago

How to Survive The Worst Launch of Your Career with Brenda Romero

Brenda Romero

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.