> Markdown version of [/videos/100311-swapping-a-data-warehouse-at-runtime-zero-downtime-migration-without-changing-a-single-client](https://www.wearedevelopers.com/videos/100311-swapping-a-data-warehouse-at-runtime-zero-downtime-migration-without-changing-a-single-client). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Imagine migrating a 620TB active data warehouse without a single maintenance window. See how a custom proxy transpiles SQL mid-flight to swap compute engines invisibly beneath millions of queries.

- **Speakers:** [Max Fischer](https://www.wearedevelopers.com/@max-fischer), [Michael O'Toole](https://www.wearedevelopers.com/@michael-o-toole)
- **Event:** World Congress 2026 Europe
- **Published:** July 10, 2026
- **Duration:** 30:28
- **URL:** https://www.wearedevelopers.com/videos/100311-swapping-a-data-warehouse-at-runtime-zero-downtime-migration-without-changing-a-single-client

## Summary

Trade Republic scaled from 3 million to 12 million customers, resulting in exponential data growth and escalating costs for their existing rigid data warehouse. Seeking decoupled open lakehouse storage with Apache Iceberg and flexible compute engines (Spark, Snowflake, Athena, DuckDB), the engineering team faced a massive hurdle: migrating a 620 TB active system without scheduling a maintenance window or demanding code rewrites from hundreds of data consumers. The solution was "Engy," a custom-built, protocol-compatible proxy that perfectly emulates the existing warehouse's wire interface, allowing users to simply change a hostname while infrastructure shifts invisibly behind the scenes.

Built on open-source foundations like SQLGlot and an optimized Rust tokenizer, Engy performs in-flight SQL transpilation, translating legacy table references, rewriting distinct dialects, and normalizing response formats before they return to the client. This abstraction layer acts as a contract that inherently decouples the pace of infrastructure evolution from consumer adoption. By processing requests at the Abstract Syntax Tree (AST) level, the team introduced semantic caching that outperforms native cloud solutions by ignoring trivial syntax variations like spacing or comments. They also layered in predictive query routing that dynamically assigns tasks to the most cost-efficient compute engine, seamlessly dropping into DuckDB for instantaneous, lightweight reads.

Owning the interface unlocked capabilities far beyond a sheer migration, turning "building the plane as we fly it" into a stable reality. Engineering shipped custom SQL operators natively into the proxy—such as an automated schema-evolving merge command or a wait function that pauses downstream dbt models until parent data arrives, bypassing the need to restart large pipeline DAGs. Further optimizations converted basic DDL commands from expensive clustered compute spin-ups into lightweight API calls. Ultimately, abstracting the database layer allowed the team to completely swap out underlying compute engines four separate times while serving 4 million daily queries with perfect, uninterrupted interop.

**Keywords:** zero-downtime data migration, data warehouse proxy pattern, in-flight sql transpilation, apache iceberg lakehouse, decoupled storage compute, abstract syntax tree caching, predictive query routing, duckdb query optimization, dbt dag orchestration, schema evolution automation, zero-copy clone emulation, open-source data infrastructure, sqlglot query parsing, wire-compatible database proxy

## Chapters

1. **Scaling data infrastructure for a rapidly growing fintech platform** (01:20) — Expanding datasets and customer transactions forced a fundamental reevaluation of legacy data architectures to support growing operational demands.
1. **Addressing data ingestion and cost challenges while scaling architectures** (02:25) — Expanding computational costs necessitated a shift from rigid daily batch jobs toward more frequent and flexible analytical ingestion.
1. **Decoupling storage and compute with open lakehouse architectures** (03:46) — Adopting standard table formats ensures independent scaling for data stores while letting teams dynamically select appropriate execution engines.
1. **Migrating moving data targets without disrupting active engineering teams** (05:03) — Abstracting the database connection behind a persistent proxy maintains application uptime during underlying infrastructure transitions.
1. **Routing queries and transpiling syntax dynamically through a proxy** (07:12) — Intercepting network traffic enables systems to safely modify connection hosts and translate execution commands directly within the request path.
1. **Demonstrating dynamic interactive querying across different execution engine backends** (10:38) — Transpilation workflows rewrite backend-specific functions automatically behind the scenes for identical queries submitted across divergent database engines.
1. **Sharing data seamlessly across multiple compute engines using custom syntax** (13:51) — Bespoke proxy commands actively align output records across different processing tools to eliminate complex intermediate persistence layers.
1. **Implementing proxy plugins to intercept schemas and database clones** (15:36) — Automating underlying configurations allows the proxy layer to silently enforce index modifications and rapidly scale basic table operations.
1. **Optimizing sequential data pipeline performance with intelligent query delays** (18:11) — Injecting conditional polling handlers at the routing tier blocks orchestration commands efficiently until dependent upstream facts resolve.
1. **Enhancing caching patterns and access controls via the proxy layer** (19:22) — Evaluating analytical queries by an abstract syntax tree unlocks precise cache hits regardless of textual differences or formatting.
1. **Extracting operational principles from processing four million daily incoming queries** (21:38) — Designing flexible interception protocols empowers vast application ecosystems to scale horizontally without introducing hard client reconfigurations.
1. **Integrating conversational algorithms directly into the query interception proxy** (24:16) — Owning the database communication tier directly supports implementing specialized conversational agents and automated query optimization metrics internally.
1. **Addressing runtime migration specifics and future open source framework ambitions** (25:11) — Audience questions probe analytical tokenization specifics and the logistical pacing behind migrating legacy schemas incrementally using automation agents.

## Related Moments

- [Summary of decoupling analytical compute and storage](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) (from "Parquet, Delta, Iceberg & Ducklake - An introduction for developers")
- [Boosting developer productivity via consolidated converged database architectures](https://www.wearedevelopers.com/videos/632-crypto-secure-data-management-with-in-database-blockchain) (from "Crypto-secure Data Management with In-Database Blockchain")
- [Solving complex platform architecture challenges at an enterprise scale](https://www.wearedevelopers.com/videos/1209-coffee-with-developers-maria-apazoglou) (from "Coffee with Developers - Maria Apazoglou")
- [Executing massive cloud network migrations while maintaining live systems](https://www.wearedevelopers.com/videos/100128-the-golden-age-of-email-owning-the-inbox-in-the-age-of-ai) (from "The Golden Age of Email: Owning the Inbox in the Age of AI")
- [Modernizing queries using microservices and federated schemas](https://www.wearedevelopers.com/videos/983-reliable-scalability-how-amazon-com-scales-on-aws) (from "Reliable scalability: How Amazon.com scales on AWS")
- [Designing data ingestion architecture with system integration](https://www.wearedevelopers.com/videos/688-implementing-an-event-sourcing-strategy-on-azure) (from "Implementing an Event Sourcing strategy on Azure")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub**
- [Staff Software Engineer](https://www.wearedevelopers.com/jobs/ext/1425755-staff-software-engineer) at **GitHub**