Distributed systems fail by default, so reliability is an architectural property that must be intentionally designed rather than treated as an add-on feature.
A slow external API combined with default timeouts can cause a system-wide outage, which is contained by setting explicit timeouts based on P99 latency.
Transient errors like timeouts or packet drops can be managed by implementing a clear strategy of retrying with exponential backoff, using a fallback, or failing fast.
Retrying operations after a timeout can cause silent duplication like double-charging, which is prevented by implementing robust idempotency using unique keys and request fingerprinting.
When a dependency is completely down, a circuit breaker prevents wasteful retries by failing fast, protecting system resources until the dependency recovers.
To avoid losing business during a persistent outage, critical operations can be decoupled using asynchronous queues, allowing the system to accept requests and process them later.
A summary of critical questions to ask during design reviews and key mental models to adopt for building resilient distributed systems from the start.
Jobs that call for the skills explored in this talk.