> Markdown version of [/videos/680-handling-incidents-collaboratively-is-like-solving-a-rubix-cube](https://www.wearedevelopers.com/videos/680-handling-incidents-collaboratively-is-like-solving-a-rubix-cube). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Handling incidents collaboratively is like solving a rubix cube

When systems crash, isolated engineering teams scramble on different sides of the Rubik's cube. Discover how collaborative incident response untangles silos and speeds up recovery.

- **Speakers:** Nele Uhlemann
- **Event:** World Congress 2023
- **Published:** September 21, 2023
- **Duration:** 24:00
- **URL:** https://www.wearedevelopers.com/videos/680-handling-incidents-collaboratively-is-like-solving-a-rubix-cube

## Summary

Engineering teams often operate on 'different sides of the Rubik's cube,' utilizing highly specialized workflows across backend development, DevOps, and Site Reliability Engineering (SRE). When a system incident occurs, this isolation creates friction. Effectively untangling the cube requires moving beyond isolated debugging to embrace collaborative incident response, spanning three essential phases: resolving, preventing, and discovering.

During the initial resolving phase, restoring functional operations quickly takes priority over identifying the root cause. This requires bringing specialized teams to a transparent, shared interface—such as collaborative incident notebooks—to establish a unified understanding of causality. Moving into the prevention phase requires transparent communication of remediation efforts across infrastructure layers. For example, if a team implements retry logic to mitigate failures, engineering groups must document whether that logic lives within a backend application or a service mesh like Istio. Duplicating or misaligning these configurations across different operational layers can cause unpredictable system behavior.

The discovery phase relies on implementing robust, standardized observability. Adopting CNCF standards like OpenTelemetry is crucial for extracting telemetry data neutrally, decoupling instrumented code from vendor lock-in. To bridge the gap between application engineers and complex SRE toolchains, open-source libraries like Autometrics enable developers to instrument code directly via simple decorators. By rendering golden signals (latency, error rate, request rate) and defining Service Level Objectives (SLOs) right inside the IDE, teams can bypass steep learning curves for querying languages like PromQL. Ultimately, surfacing telemetry directly within developer environments aligns software creators with operations teams, integrating tools like Prometheus and Grafana without forcing engineers out of their native workflows.

**Keywords:** incident response collaboration, cross-functional engineering, observability best practices, open telemetry adoption, service level objectives (SLOs), vendor neutral telemetry, golden signals monitoring, autometrics instrumentation, prometheus and grafana dashboards, distributed tracing causality, service mesh retry logic, promql queries, ide metrics integration, reliability engineering workflows, collaborative incident notebooks

## Chapters

1. **Navigating specialized roles and toolsets across engineering teams** (00:00) — Incident resolution requires cross-functional collaboration because backend developers, SREs, and DevOps professionals primarily operate within domain-specific workflows.
1. **Creating a shared understanding to resolve incidents quickly** (03:50) — Centralized platforms like shared notebooks establish data transparency to help diverse teams align on causality and fast fixes.
1. **Preventing future incidents through proper retry strategies** (06:37) — Establishing cross-team best practices determines whether network retries belong in the service mesh or the backend application codebase.
1. **Discovering incidents using logs, metrics, and traces** (09:02) — Instrumenting application code to expose continuous telemetry data allows operational teams to detect latency outliers and trace root causes.
1. **Standardizing telemetry extraction using the OpenTelemetry framework** (10:59) — Adopting a vendor-neutral standard simplifies application instrumentation while allowing infrastructure teams to switch backend storage layers effortlessly.
1. **Generating instant observability metrics with the Autometrics library** (13:49) — Function-level code decorators automatically produce latency, error, and request rate metrics without requiring developers to leave their IDE.
1. **Instrumenting a Python backend application with Prometheus metrics** (15:29) — Applying the Autometrics decorator to a FastAPI endpoint automatically exposes standard metrics for local querying and developer inspection.
1. **Defining programmatic service level objectives for specific endpoints** (18:30) — Attaching success rate objectives directly in application code translates logic-level expectations into actionable alerts on operational dashboards.
1. **Balancing tool consolidation and collaborative open source workflows** (20:57) — Embracing specific technologies for unique use cases while maintaining shared communication standards fosters effective cross-team incident management.

## Related Moments

- [Integrating service level objectives into incident management](https://www.wearedevelopers.com/videos/854-serverless-observability-where-slos-meet-transforms) (from "Serverless Observability: where SLOs meet transforms")
- [Exploring advanced observability stacks and distributed infrastructure challenges](https://www.wearedevelopers.com/videos/544-plan-ci-cd-on-the-enterprise-level) (from "Plan CI/CD on the Enterprise level!")
- [Applying software development methodologies to incident response](https://www.wearedevelopers.com/videos/101-applying-agile-principles-to-incident-management) (from "Applying Agile Principles to Incident Management ")
- [Streamlining incident response and root cause analysis automatically](https://www.wearedevelopers.com/videos/1539-agentic-devops-how-ai-powered-automation-transforms-software-delivery-on-github-and-azure) (from "Agentic DevOps: How AI-Powered Automation Transforms Software Delivery on GitHub and Azure")
- [Leveraging scope topology for systemic incident troubleshooting](https://www.wearedevelopers.com/videos/1618-planet-scale-dashboards) (from "Planet-Scale Dashboards")
- [Engaging software developers deeply in secure engineering practices](https://www.wearedevelopers.com/videos/193-building-security-champions) (from "Building Security Champions")

## Related Articles

- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)
- [Effortlessly Scale Prometheus With The Telemetry Data Platform – And Keep your Grafana Dashboards, Too!](https://www.wearedevelopers.com/magazine/3-effortlessly-scale-prometheus-with-the-telemetry-data-platform-and-keep-your-grafana-dashboards-too)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)

## Related Jobs

- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [Senior Software Engineer, Enterprise Products](https://www.wearedevelopers.com/jobs/ext/1841248-senior-software-engineer-enterprise-products) at **GitHub**
- [Senior Software Engineer, Client Apps Platform](https://www.wearedevelopers.com/jobs/ext/1773893-senior-software-engineer-client-apps-platform) at **GitHub**
- [Senior Security Engineer, Incident Response](https://www.wearedevelopers.com/jobs/ext/1347114-senior-security-engineer-incident-response) at **Twilio**
- [Engineer, Offensive Security Organization](https://www.wearedevelopers.com/jobs/ext/1992296-engineer-offensive-security-organization) at **Twilio**