WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2022 • Jun 15, 2022

I broke the production

Arto Liukkonen

A new hire brought down production with a rogue script. Discover why resilient engineering teams embrace blameless post-mortems to fortify systems instead of hunting for a scapegoat.

Pause
Mute Enter Fullscreen
#1 about 3 min

Recognizing the inevitability of breaking production environments

Sharing early career failures normalizes the reality that all developers will eventually cause widespread system chaos.

#2 about 3 min

Managing constant production updates at massive data scale

Processing petabytes of advertising data requires tightly managed continuous deployment pipelines supporting dozens of daily updates.

#3 about 3 min

Triggering unexpected outages during restrictive data backfills

Attempting massive data backfills directly in production due to compliance restrictions can easily trigger severe monitoring spikes.

#4 about 3 min

Reconciling individual actions with good technical intentions

Overcoming the human tendency to judge coworkers by actions rather than intentions fosters fairer incident assessments.

#5 about 3 min

Recognizing systemic flaws over pointing fingers at developers

Treating deployment outages as failures of the pull request process rather than individual mistakes prevents toxic blame loops.

#6 about 2 min

Fostering self awareness to prevent toxic team behaviors

Cultivating personal responsibility and empathy stops frustration from cascading downward into harmful workplace chain reactions.

#7 about 2 min

Implementing blameless postmortems to strengthen system resilience

Borrowing practices from the healthcare and aviation industries turns every technical mistake into an opportunity for infrastructure improvement.

#8 about 4 min

Applying positive feedback ratios to technical code reviews

Leverging a five-to-one positive feedback model during pull requests builds team trust and breaks the cycle of purely critical interactions.

#9 about 2 min

Anticipating systemic failures using proactive premortem exercises

Simulating disaster scenarios before code ships helps uncover hidden edge cases and vulnerabilities within complex architectures.

#10 about 4 min

Embracing organizational learning following stressful production incidents

Navigating extended recovery windows proves that eliminating blame allows teams to focus entirely on rapid system restoration.

#11 about 3 min

Handling customer impact and mitigating third party failures

Buffering incidents through customer support provides necessary space for engineers while past external routing failures offer valuable historical lessons.

Matching moments

2:08 min

Navigating and mitigating the impacts of broken engineering cultures

Oskar Kruschitz Oskar Kruschitz · Europe 2026 Virtual

1:49 min

The feedback loop of leadership choices and technical outcomes

Oskar Kruschitz Oskar Kruschitz · Europe 2026 Virtual

3:03 min

Building a corporate culture that actively celebrates failure

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

3:34 min

Writing blameless and detailed incident postmortems

Martin Beránek · LIVE

1:54 min

Embracing failures and analyzing high-profile software engineering mistakes

Christian Seifert · WWC 2022

1:02 min

Establishing a constructive failure culture for continuous learning

Rudi Bauer Rudi Bauer +1 · Cappuccino with HR

More from World Congress 2022

What the heck do "declarative" and "reactive" actually mean? thumbnail
30:40
almost 4 years ago

What the heck do "declarative" and "reactive" actually mean?

André Kovac

The Design and Evolution of C++ thumbnail
1:01:15
about 4 years ago

The Design and Evolution of C++

Bjarne Stroustrup

But, you're not Facebook thumbnail
31:12
about 4 years ago

But, you're not Facebook

Kristijan Ristovski

Communicate efficiently with Software Architecture Diagrams thumbnail
28:11
about 4 years ago

Communicate efficiently with Software Architecture Diagrams

Simon Lasselsberger

Get Ready for Jakarta EE 10! thumbnail
42:53
about 4 years ago

Get Ready for Jakarta EE 10!

Ivar Grimstad

Keep your code refactorable thumbnail
26:46
about 4 years ago

Keep your code refactorable

Gerrit Stapper

Upcoming sessions on this topic

Open session

World Congress 2026 North America

It passed auth, then production caught fire

Alex Olivier

Co-founder & CPO @ Cerbos | OpenID AuthZEN Co-chair

Alex Olivier
Open session

World Congress 2026 North America

The Geometry of Incidents: What User-Impact Shapes Reveal About Platform Architecture

Bala Subrahmanyam Kambala

Staff Platform Engineer at Oracle Cloud Infrastructure

Bala Subrahmanyam Kambala
Open session

World Congress 2026 North America

Culture Doesn't Scale Itself: Leading Engineering Teams Through Hypergrowth and the AI Transition

Thanos Baskous

VP of Engineering and Co-founder of Cogent Security

Thanos Baskous
Open session

World Congress 2026 North America

The Autonomous Performance Agent: A Netflix Production Story

Rajat Shah

Staff Software Engineer, AI Platform, Netflix

Rajat Shah
Open session

World Congress 2026 North America

Evals Are Infra: Building AI Systems Developers Can Actually Trust

Phoebe Wang

Member of Technical Staff at OpenAI

Phoebe Wang
Open session

World Congress 2026 North America

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Garvit Kataria, Sahil Sabharwal

Garvit Kataria
Sahil Sabharwal

Related videos

Blameless Postmortems That Change Nothing: Cultural Anti-Patterns in Incident Learning thumbnail
28:02
WWC Europe 2026 • about 1 month ago

Blameless Postmortems That Change Nothing: Cultural Anti-Patterns in Incident Learning

Daniel Schley

Empathy: The secret sauce of Resilience thumbnail
29:55
WWC 2022 • about 4 years ago

Empathy: The secret sauce of Resilience

Malin Litwinski

Answering the Million Dollar Question: Why did I Break Production? thumbnail
29:38
WWC 2024 • almost 2 years ago

Answering the Million Dollar Question: Why did I Break Production?

Luís Ventura

Fail fast. How embracing failure leads to innovation at ZEISS. thumbnail
27:40
WWC 2024 • almost 2 years ago

Fail fast. How embracing failure leads to innovation at ZEISS.

Katharina Enßle

From Code to Culture: Why Leadership Determines Software Quality thumbnail
30:07
Europe 2026 Virtual • about 1 month ago

From Code to Culture: Why Leadership Determines Software Quality

Oskar Kruschitz

How to Survive The Worst Launch of Your Career with Brenda Romero thumbnail
30:53
WWC 2023 • almost 3 years ago

How to Survive The Worst Launch of Your Career with Brenda Romero

Brenda Romero

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.