Skip to content

Session

The Geometry of Incidents: What User-Impact Shapes Reveal About Platform Architecture

with Bala Subrahmanyam Kambala

About This Session

Incidents are usually reviewed as timelines: what failed, who owned it, and how we restored service. That works well for understanding a single outage. But when you operate platforms used by many services, the root causes change while the user-impact patterns start to look familiar. This talk introduces incident shapes: a way to look at failures by the pattern they draw across impact, time, and blast radius. I’ll use a few public incidents as reference points, including CrowdStrike’s outage, AWS’s DynamoDB outage where retry amplification played a role, Cloudflare’s WAF incident, and GitHub’s 2018 database failover incident. The incident shapes are useful because they change the questions we ask. A sudden spike makes us look at rollout containment and rollback paths. A slow burn pushes us to examine saturation, queues, and retries. A repeating sawtooth suggests the system may be recovering temporarily without becoming stable. Fan-out patterns expose the risk of shared platform layers. Boundary shifts are often the hardest to catch: one layer reports success, while users are still having a bad experience. The main idea is simple: the shape of user impact can tell us what the what the architecture failed to protect against. Attendees will learn how to quantify impact using breadth, depth, and duration, and how to connect those shapes to engineering responses such as staged rollouts, rollback automation, retry budgets, semantic canaries, cell isolation, contract checks, and end-to-end verification. The goal is to make postmortems more useful: not just to explain what happened, but to help design platforms that are harder to break in the same way twice.

Topics

  • DevOps
  • Distributed Systems
  • Infrastructure
  • Observability
  • Site Reliability Engineering (SRE)
  • Software Architecture