Your Rollback Plan is Too Slow and How to Fix It

Back in 2025, two of the most mature engineering organisations on the planet went down, with giants Cloudflare and Google Cloud suffering global outages. Interestingly, postmortems into both reached the very similar conclusions that some have overlooked in the time since.

In June of 2025, Google Cloud suffered a global outage after a backend binary change propagated across the platform, noting in the postmortem that if the change had been behind a feature flag, it would have been caught in staging before it could have a global impact. Then, in November, Cloudflare lost more than five hours to a routine configuration update, with the company committing to enable more global switches after the issue was resolved.

It’s important to realise that neither of these was a deployment failure. Both deployments worked: the pipelines were green, the infrastructure was healthy, and the systems were running exactly as instructed. What was missing was a way to undo the change without deploying again.

That’s the part I want to talk about, because I think most teams have a rollback (or roll-forward) plan that only looks like a plan.

“Revert and redeploy” is not a recovery plan

If I ask teams how they recover from a bad release, I usually get some version of this process: (a) Revert the commit (b) Run the pipeline and © Wait for it to roll out.

Let’s consider this for a second. The kind of issue we’re discussing usually happens at 2am, with someone who didn’t write the change trying to fix it. What needs to happen in practice? A long-running CI pipeline on a monorepo. A few artefacts to rebuild. Rolling restarts across regions. Some cache warm-up.

If you’re lucky, it takes fifteen minutes. On a bad day, it’s an hour. For some unlucky teams, it’s over 24 hours. And the revert itself may need reverting too, because your main branch has already changed three times since the original deployment.

Moreover, this is all based on the assumption that you know which change caused the problem. If multiple services were deployed within the same window, locating the issue to begin with can make the issue even more complex, and stressful for the team working on it.

Early in my career, at a startup, shipping a new homepage meant picking a quiet evening, deploying, and then sitting there watching dashboards with the rollback command already typed into a terminal, ready to go. The stakes were tiny: a few thousand users and a ruined evening. Scale that same ritual up to an organisation losing hundreds of thousands of dollars per hour, and you can see why big companies try to reduce how often they deploy.

Another important point: none of this is a DevOps failure. DevOps did its job: the code got to production reliably and repeatedly. The gap is what happens next: how the code behaves once it’s running, and how quickly you can change your mind. There’s a name gaining traction for the practices that close that gap: FeatureOps, and I’ll use it here for convenience.

AI is not the cause, but it is the accelerant

Almost every developer I talk to now has AI somewhere in their workflow, and the productivity story is real. The stability story, however is less flattering. DORA’s research has found that, as AI assistance increases, delivery stability tends to decrease by as much as 7%.

In other words: more changes, produced faster, reviewed less thoroughly, often by someone who didn’t write them and can’t always explain them. And in 2026, it’s not just about writing code, as agents are acting autonomously inside production systems too.

Of course, none of this changes the fundamentals. But it does shorten the time between “we shipped something” and “we need to turn that off”.

What reversibility actually looks like

Four things, roughly in order of how much they buy you:

Deploy without releasing. You deploy a new feature, but it stays switched off. Turning it on becomes a separate business decision that you can make later, when you (or your customers) are ready.

Cap the blast radius. You enable the new behaviour for just 1% of traffic to begin with. Perhaps you start with internal users, then beta testers, then everyone else. This is a game changer because a bad change that affects only beta testers or 1% of traffic is an incident you can discuss calmly.

Reverse without deploying. You undo the release with a simple runtime configuration change, rather than running the pipeline again. That should take seconds, not hours or days.

Don’t forget governance. You should get an audit log of who changed what, when, and for whom, as standard. And no individual should be able to change production settings without peer review, just as they would need for code. Skip this, and you’ve replaced a slow, auditable release process with a fast, invisible one. You want your compliance people to sleep at night too.

Now the warts

Everything above describes the happy path. Let’s also cover the things that tend not to make it into the conference talk.

Flags rot. Every flag introduces a branch in your code that somebody eventually has to remove. Nobody deletes them because, unfortunately, it’s not a particularly fun job. Two years later, you have hundreds of flags, half of them permanently enabled, and nobody remembers which flag controls which feature. Sadly, a rollback lever you’re afraid to touch isn’t much of a rollback lever.

The test matrix explodes. Ten independent flags mean 1,024 possible states. But you’re not going to test all 1,024. You’re testing the two you thought of, while production is running one of the other 1,022.

Runtime config becomes its own outage surface. This is an uncomfortable reality, given that Cloudflare’s outage was itself caused by a configuration change. Moving risk out of the deployment pipeline and into runtime config doesn’t eliminate it; it often just moves it somewhere with less review.

If you want a real historical example, Knight Capital is the textbook case of what can go wrong. In 2012, the company reused an old flag to activate new code, but one of its servers hadn’t received the deployment. The flag was switched on. On that server, it activated eight-year-old dead code, costing the company roughly $440 million in 45 minutes.

Doing it anyway, but carefully

I still think reversibility is worth the cost. The alternative is being unable to stop a change you’ve already shipped, and 2025 showed us twice what that can cost.

So, what are the technical and organisational best practices for runtime control?

• Give every flag a type and an expiry date. A release flag that outlives its rollout is a bug.
• Make cleanup part of the definition of done. Not a quarterly cleanup sprint that never happens. The ticket isn’t closed until the flag is gone.
• Keep permanent flags few and obvious. Kill switches and operational toggles are legitimately long-lived; everything else should be temporary.
• Treat flag changes as production changes. Maintain an audit trail, require approvals, and give them the same level of scrutiny as a deployment.
• Test the switch, not just the feature. Don’t forget to test the off path, because that’s the one you’ll need under pressure.

The question worth asking your team

Forget tooling for a moment and ask your team one thing: the last time something went wrong in production, how long was it between “this is bad” and “this is off”?

If the honest answer is measured in pipeline runs, that’s the real gap to address.