Nele Uhlemann

Handling incidents collaboratively is like solving a rubix cube

What if developers could instrument their code and define SLOs with a simple decorator? Learn a new approach to making observability a shared responsibility.

Handling incidents collaboratively is like solving a rubix cube
#1about 4 minutes

The Rubik's Cube metaphor for engineering teams

Different engineering teams like backend and SREs operate on different sides of the system, requiring collaboration during incidents.

#2about 3 minutes

The first phase of resolving incidents collaboratively

The initial step in incident response is to establish a common understanding and transparency across teams before applying quick fixes.

#3about 2 minutes

Preventing future incidents with best practices

After resolving an incident, teams must collaborate on prevention by documenting best practices for patterns like service retries.

#4about 2 minutes

Discovering incidents through system observability

The discovery phase relies on making systems observable by collecting telemetry data like logs, metrics, and traces.

#5about 2 minutes

Standardizing telemetry collection with OpenTelemetry

OpenTelemetry provides a vendor-neutral standard for instrumenting applications, preventing vendor lock-in for observability backends.

#6about 2 minutes

Simplifying metrics with the Autometrics library

The open-source Autometrics library uses decorators to automatically generate key metrics like latency, errors, and request rate from functions.

#7about 5 minutes

Demo of generating metrics and SLOs from code

A live demo shows how Autometrics provides live metrics in the IDE and helps define SLOs that can be visualized in Grafana.

#8about 1 minute

Summary of collaborative incident management phases

A recap of the three key phases for collaborative incident handling: resolving, preventing, and discovering issues together.

#9about 2 minutes

Q&A on tooling and open source contribution

The speaker answers audience questions about managing tool complexity and the role of community contributions in open-source projects.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Applying agile and SRE principles to incident response
06:30 MIN

Applying agile and SRE principles to incident response

Applying Agile Principles to Incident Management

Actionable takeaways for SREs on incident management
27:09 MIN

Actionable takeaways for SREs on incident management

Serverless Observability: where SLOs meet transforms

Using an incident console to manage response and resolvers
20:29 MIN

Using an incident console to manage response and resolvers

Applying Agile Principles to Incident Management

Fostering cross-team collaboration with SLOs
24:30 MIN

Fostering cross-team collaboration with SLOs

Serverless Observability: where SLOs meet transforms

How engineers handle production errors and monitoring
29:58 MIN

How engineers handle production errors and monitoring

DevOps at Netflix

Overcoming observability challenges with a unified platform
18:09 MIN

Overcoming observability challenges with a unified platform

All your telemetry data from any source in one place

Improving incident response to make on-call less painful
28:08 MIN

Improving incident response to make on-call less painful

What Developers Get Wrong About Application Quality

Building resilience across all software stack layers
13:39 MIN

Building resilience across all software stack layers

System Resilience: Surviving the Software Storm

Featured Partners

Related Videos

See all videos
Empathy: The secret sauce of Resilience29:55

Empathy: The secret sauce of Resilience

Malin Litwinski

Unveiling the Dark Side: Navigating the Pitfalls of Digital Ambitions21:39

Unveiling the Dark Side: Navigating the Pitfalls of Digital Ambitions

Johannes Hansen

One size fits all! Not at all!59:00

One size fits all! Not at all!

Ixchel Ruiz

The Software Bug All Stars - and what we can learn from them27:27

The Software Bug All Stars - and what we can learn from them

Christian Seifert

I broke the production26:18

I broke the production

Arto Liukkonen

Applying Agile Principles to Incident Management 37:33

Applying Agile Principles to Incident Management

Tobias Dunn-Krahn

Mastering AI-Driven Problem Solving in Engineering with Observability27:39

Mastering AI-Driven Problem Solving in Engineering with Observability

Jemiah Sius

The AI-Ready Stack: Rethinking the Engineering Org of the Future28:09

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Jan Oberhauser, Mirko Novakovic, Alex Laubscher & Keno Dreßel

From learning to earning

Jobs that call for the skills explored in this talk.

Platform Engineers

Incident

Remote
Terraform
Google Cloud Platform
Continuous Integration