> Markdown version of [/videos/1065-chaos-in-containers-unleashing-resilience?t=698](https://www.wearedevelopers.com/videos/1065-chaos-in-containers-unleashing-resilience?t=698). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Chaos in Containers - Unleashing Resilience

Traditional IaC tools execute once and ignore state drift. Shift to continuous reconciliation with Crossplane to automatically heal external infrastructure and achieve true cloud resilience.

- **Speakers:** [Maish Saidel-Keesing](https://www.wearedevelopers.com/@maish-saidel-keesing)
- **Event:** World Congress 2024
- **Published:** August 20, 2024
- **Duration:** 27:52
- **URL:** https://www.wearedevelopers.com/videos/1065-chaos-in-containers-unleashing-resilience

## Summary

Modern cloud environments are often plagued by an overreliance on rigid CI/CD pipelines for infrastructure lifecycle management. While traditional Infrastructure as Code (IaC) tools like Terraform act as efficient "one-shot executors" that translate code into API requests, they fail to prevent state drift because they do not continuously monitor resources. To achieve true cloud resilience, operations must shift away from single-execution deployment scripts toward continuous reconciliation systems that mirror hyperscaler control planes.

By leveraging Kubernetes as a universal control plane powered by Crossplane, engineering teams can continuously orchestrate, abstract, and manage external infrastructure natively. This model enables developers to unify application code and infrastructure constraints; they can package cloud services—such as an AWS API Gateway or a DynamoDB database—directly inside standard Helm charts. Managed via GitOps methodologies using ArgoCD, Crossplane evaluates real-world states against declared Git configurations and automatically rectifies drift. A major underlying insight is that platform teams can design "composite resources," which abstract complex, multi-resource cloud environments into single, digestible Kubernetes objects, fully utilizing the native ecosystem for validation and secrets management.

A primary structural challenge with this cloud-native paradigm is the "chicken-or-egg" bootstrapping sequence: a cluster is required to run Crossplane, which itself provisions clusters. This is resolved by initializing a lightweight, temporary Kubernetes instance to deploy Crossplane, using it to spin up the permanent control plane, and seamlessly transferring the state via backup and restore. Crucially, because this control plane handles zero production workloads and exists purely for orchestration, it remains entirely ephemeral—meaning worst-case failures merely interrupt observability and can be recreated on demand. For teams heavily invested in legacy configuration tools, Crossplane's Terraform provider also offers a pragmatic stepping stone to wrap and orchestrate existing modules natively within Kubernetes.

**Keywords:** kubernetes control plane, crossplane, continuous reconciliation, gitops workflows, argocd deployment, infrastructure state drift, custom composite resources, helm chart provisioning, ephemeral control planes, IaC limitations, cloud orchestration, terraform migration, AWS infrastructure management, CI/CD pipeline bottlenecks, infrastructure lifecycle management

## Chapters

1. **Managing infrastructure chaos with endless deployment pipelines** (00:03) — Traditional continuous deployment pipelines lack proper lifecycle management for complex cloud infrastructure.
1. **Evolution from manual servers to cloud configuration APIs** (02:44) — The shift from script-based machine management to API-centric cloud service orchestration completely changed infrastructure capabilities.
1. **Infrastructure drift risks with point-in-time execution tools** (04:48) — Point-in-time executors translate code into static API requests without continuous reconciliation to prevent configuration drift.
1. **Extending the Kubernetes control plane with custom resources** (07:46) — Crossplane provides continuous reconciliation by packaging external cloud API objects directly into custom Kubernetes composite resources.
1. **Synchronizing configurations through Git repository tracking operators** (11:38) — Tracking infrastructure state in version control allows operators like ArgoCD to continuously sync manifests directly to the cluster.
1. **Provisioning full infrastructure environments from custom helm charts** (13:48) — Embedding custom resource specifications inside standard manifest files seamlessly provisions managed cloud databases and external networking rules.
1. **Bootstrapping an ephemeral control plane cluster with Crossplane** (19:18) — Temporarily spinning up lightweight nodes makes it possible to initialize a primary control cluster that subsequently manages its own underlying infrastructure.
1. **Handling state recovery and migrating from legacy configurations** (22:34) — Technical discussions examine complex resource schema versioning, restoring disconnected infrastructure references, and leveraging native provider wrappers for module migrations.

## Related Moments

- [Provisioning component dependencies using Crossplane and Kubernetes manifests](https://www.wearedevelopers.com/videos/1197-dapr-in-practice) (from "Dapr in Practice")
- [Managing the operational complexity of modern cloud infrastructure configurations](https://www.wearedevelopers.com/videos/413-the-future-of-cloud-is-abstraction-why-kubernetes-is-not-the-endgame-for-stackit) (from "The Future of Cloud is Abstraction - Why Kubernetes is not the Endgame for STACKIT ")
- [Transitioning from monolith architectures to microservices and Kubernetes](https://www.wearedevelopers.com/videos/108-get-ready-for-operations-by-pull-requests) (from "Get ready for operations by pull requests")
- [Managing deployments with Kubernetes and Terraform self-healing](https://www.wearedevelopers.com/videos/531-implementing-feature-environments-with-aws-and-terraform) (from "Implementing Feature Environments with AWS and Terraform")
- [Orchestrating with Kubernetes against Docker and accessing slides](https://www.wearedevelopers.com/videos/487-debug-a-kubernetes-operator) (from "Debug a Kubernetes Operator")
- [Exploring advanced observability stacks and distributed infrastructure challenges](https://www.wearedevelopers.com/videos/544-plan-ci-cd-on-the-enterprise-level) (from "Plan CI/CD on the Enterprise level!")

## Related Articles

- [Learning Kubernetes made easy with KubeCampus](https://www.wearedevelopers.com/magazine/348-learning-kubernetes-made-easy-with-kubecampus)
- [Building AI Solutions with Rust and Docker](https://www.wearedevelopers.com/magazine/494-building-ai-solutions-with-rust-and-docker)
- [How we Build The Software of Tomorrow](https://www.wearedevelopers.com/magazine/120-how-we-build-the-software-of-tomorrow)
- [Now is the time for industrialized software development](https://www.wearedevelopers.com/magazine/601-now-is-the-time-for-industrialized-software-development)

## Related Jobs

- [Platform Engineer (DevOps)](https://www.wearedevelopers.com/jobs/48264-platform-engineer-devops) at **WDW Consulting GmbH**
- [Lead Cloud DevSecOps Engineer - Kubernetes](https://www.wearedevelopers.com/jobs/ext/1659167-lead-cloud-devsecops-engineer-kubernetes) at **BWI GmbH**
- [Senior Cloud Native Solution Architect (all genders welcome) - Kubernetes, CNCF, MlOps](https://www.wearedevelopers.com/jobs/ext/101488-senior-cloud-native-solution-architect-all-genders-welcome-kubernetes-cncf-mlops) at **Rosenxt Group**
- [Senior Cloud Native Solution Architect (all genders welcome) - Kubernetes, CNCF, MlOps](https://www.wearedevelopers.com/jobs/ext/66342-senior-cloud-native-solution-architect-all-genders-welcome-kubernetes-cncf-mlops) at **Rosenxt Group**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**
- [Cloud-Native Architect (all genders welcome) - Kubernetes, CNCF, MLOps](https://www.wearedevelopers.com/jobs/ext/101479-cloud-native-architect-all-genders-welcome-kubernetes-cncf-mlops) at **Rosenxt Group**