> Markdown version of [/videos/426-adjusting-pod-eviction-timings-in-kubernetes](https://www.wearedevelopers.com/videos/426-adjusting-pod-eviction-timings-in-kubernetes). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Adjusting Pod Eviction Timings in Kubernetes

Default Kubernetes eviction timings kill stateful app availability. Learn how to combine aggressive failover tolerations with container-native storage to instantly migrate volumes during node failures.

- **Speakers:** Andrew Pruski
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 29:06
- **URL:** https://www.wearedevelopers.com/videos/426-adjusting-pod-eviction-timings-in-kubernetes

## Summary

Running single-pod stateful applications like SQL Server in Kubernetes introduces unique high-availability challenges, particularly during node failures. By default, Kubernetes waits five minutes before evicting a pod from an unresponsive node. While this behavior is perfectly adequate for stateless applications that distribute traffic across multiple replicas, it results in unacceptable downtime for stateful databases constrained to a single pod. To maintain stringent service level agreements, administrators must actively modify these default cluster behaviors.

To accelerate pod failover, developers can customize deployment tolerations, specifically reducing the `not-ready` and `unreachable` node timeouts to as little as 10 seconds. However, rapidly spinning up a replacement pod exposes a secondary infrastructure bottleneck surrounding persistent storage. In managed environments like Azure Kubernetes Service (AKS), default storage classes leave the persistent volume bound to the downed node. Consequently, the newly scheduled pod hangs indefinitely in a container creating state, paralyzed by multi-attach errors because the underlying storage cannot gracefully detach from the unresponsive host.

Achieving true high availability for stateful Kubernetes workloads requires pairing aggressive pod eviction timings with responsive, container-native storage. Integrating third-party storage solutions like Portworx establishes custom storage classes capable of instantly migrating persistent volume claims across nodes. By combining rapid pod tolerations with automated storage detachment, operations teams can engineer robust failover pipelines for legacy databases, avoiding extended outages and minimizing disruptive on-call alerts.

**Keywords:** kubernetes pod eviction, stateful workloads, sql server containers, azure kubernetes service, deployment tolerations, persistent volume claims, multi-attach errors, container creating state, portworx storage class, database high availability, kubernetes node timeouts, storage detachment, virtual machine scale sets, container-native storage

## Chapters

1. **Running stateful database applications in container environments** (00:05) — Background on deploying SQL Server in containers to accelerate environment refresh processes.
1. **Default pod eviction behavior during node failures** (03:11) — Testing default node failure behavior reveals a five-minute downtime before pods reschedule.
1. **Adjusting pod eviction timeouts with deployment tolerations** (10:46) — How adding unready and unreachable tolerations to deployments reduces pod recreation time to ten seconds.
1. **Storage attachment errors during stateful pod failover** (15:10) — Default storage classes block rapid failover because persistent volumes remain attached to the downed node.
1. **Enabling rapid storage failover with third-party storage classes** (21:26) — Implementing dedicated storage solutions allows persistent volumes to detach and reattach instantly during node failures.
1. **Summary of stateful high availability requirements in clusters** (28:12) — Proper toleration timings and advanced storage management are necessary for reliable stateful application failover.

## Related Moments

- [Managing stateful application data with persistent volume claims](https://www.wearedevelopers.com/videos/530-mastering-kubernetes-beginner-edition) (from "Mastering Kubernetes – Beginner Edition")
- [Migrating stateful applications between Kubernetes clusters](https://www.wearedevelopers.com/videos/425-it-s-all-about-the-data) (from "It's all about the Data")
- [Challenges of deploying stateful databases on native Kubernetes](https://www.wearedevelopers.com/videos/74-databases-on-kubernetes-why-you-should-care) (from "Databases on Kubernetes: Why you should care")
- [Maintaining availability through stateless scaling and redundancy](https://www.wearedevelopers.com/videos/511-our-journey-with-spring-boot-in-a-microservice-architecture) (from "Our journey with Spring Boot in a microservice architecture")
- [Addressing pod scaling speeds and event failure handling](https://www.wearedevelopers.com/videos/243-serverless-native-java-with-quarkus) (from "Serverless-Native Java with Quarkus")
- [Optimizing persistent storage performance for containerized stateful applications](https://www.wearedevelopers.com/videos/74-databases-on-kubernetes-why-you-should-care) (from "Databases on Kubernetes: Why you should care")

## Related Articles

- [Learning Kubernetes made easy with KubeCampus](https://www.wearedevelopers.com/magazine/348-learning-kubernetes-made-easy-with-kubecampus)
- [Building AI Solutions with Rust and Docker](https://www.wearedevelopers.com/magazine/494-building-ai-solutions-with-rust-and-docker)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [What does the history of data storage tell us about the future?](https://www.wearedevelopers.com/magazine/495-what-does-the-history-of-data-storage-tell-us-about-the-future)

## Related Jobs

- [Staff Software Engineer, Compute Platform](https://www.wearedevelopers.com/jobs/ext/2615501-staff-software-engineer-compute-platform) at **GitHub**
- [Senior Software Engineer, Infrastructure](https://www.wearedevelopers.com/jobs/48459-senior-software-engineer-infrastructure) at **Docker, Inc.**
- [Principal Backend Engineer, Hub](https://www.wearedevelopers.com/jobs/48449-principal-backend-engineer-hub) at **Docker, Inc.**
- [Principal Software Engineer, Docker Hardened Images](https://www.wearedevelopers.com/jobs/48453-principal-software-engineer-docker-hardened-images) at **Docker, Inc.**
- [Senior Manager, Engineering, Local Runtime](https://www.wearedevelopers.com/jobs/48455-senior-manager-engineering-local-runtime) at **Docker, Inc.**
- [Cloud SecOps Engineer - Storage](https://www.wearedevelopers.com/jobs/ext/3323312-cloud-secops-engineer-storage) at **BWI GmbH**