WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 10, 2026 • Session details

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Max Fischer , Michael O'Toole

Imagine migrating a 620TB active data warehouse without a single maintenance window. See how a custom proxy transpiles SQL mid-flight to swap compute engines invisibly beneath millions of queries.

Pause
Mute Enter Fullscreen
#1 about 2 min

Scaling data infrastructure for a rapidly growing fintech platform

Expanding datasets and customer transactions forced a fundamental reevaluation of legacy data architectures to support growing operational demands.

#2 about 2 min

Addressing data ingestion and cost challenges while scaling architectures

Expanding computational costs necessitated a shift from rigid daily batch jobs toward more frequent and flexible analytical ingestion.

#3 about 2 min

Decoupling storage and compute with open lakehouse architectures

Adopting standard table formats ensures independent scaling for data stores while letting teams dynamically select appropriate execution engines.

#4 about 3 min

Migrating moving data targets without disrupting active engineering teams

Abstracting the database connection behind a persistent proxy maintains application uptime during underlying infrastructure transitions.

#5 about 4 min

Routing queries and transpiling syntax dynamically through a proxy

Intercepting network traffic enables systems to safely modify connection hosts and translate execution commands directly within the request path.

#6 about 4 min

Demonstrating dynamic interactive querying across different execution engine backends

Transpilation workflows rewrite backend-specific functions automatically behind the scenes for identical queries submitted across divergent database engines.

#7 about 2 min

Sharing data seamlessly across multiple compute engines using custom syntax

Bespoke proxy commands actively align output records across different processing tools to eliminate complex intermediate persistence layers.

#8 about 3 min

Implementing proxy plugins to intercept schemas and database clones

Automating underlying configurations allows the proxy layer to silently enforce index modifications and rapidly scale basic table operations.

#9 about 2 min

Optimizing sequential data pipeline performance with intelligent query delays

Injecting conditional polling handlers at the routing tier blocks orchestration commands efficiently until dependent upstream facts resolve.

#10 about 3 min

Enhancing caching patterns and access controls via the proxy layer

Evaluating analytical queries by an abstract syntax tree unlocks precise cache hits regardless of textual differences or formatting.

#11 about 3 min

Extracting operational principles from processing four million daily incoming queries

Designing flexible interception protocols empowers vast application ecosystems to scale horizontally without introducing hard client reconfigurations.

#12 about 1 min

Integrating conversational algorithms directly into the query interception proxy

Owning the database communication tier directly supports implementing specialized conversational agents and automated query optimization metrics internally.

#13 about 6 min

Addressing runtime migration specifics and future open source framework ambitions

Audience questions probe analytical tokenization specifics and the logistical pacing behind migrating legacy schemas incrementally using automation agents.

Matching moments

1:21 min

Summary of decoupling analytical compute and storage

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

4:56 min

Boosting developer productivity via consolidated converged database architectures

Wei Hu Wei Hu · World Congress 2023

2:14 min

Solving complex platform architecture challenges at an enterprise scale

Maria Apazoglou · Coffee With Developers

2:11 min

Executing massive cloud network migrations while maintaining live systems

Dana Lawson Dana Lawson +1 · World Congress 2026 Europe

4:30 min

Modernizing queries using microservices and federated schemas

Florian Mair Florian Mair · World Congress 2024

3:00 min

Designing data ingestion architecture with system integration

Eldert Grootenboer +1 · World Congress 2023

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
3 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
3 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
3 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
3 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
3 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
3 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

How We Cut Our API's p99 Latency from Minutes to Under a Second

  • Deepak Agrawal

    Atlassian

    Principal Software Engineer

Open session

We Deleted Our Observability Stack and Rebuilt It with OTEL: 12 Engineers to 4 at 20K+ clusters

  • Yash Sharma

    DigitalOcean

    Developer Advocate

Open session

Architecting Trusted AI: Coupling Multi-Agent Systems with dbt Semantic Store for Zero-Hallucination

  • Pushkar Mishra

    J.P. Morgan

    Senior Vice President

Open session

Multi-Cloud Isn't a Project. It's a Side Effect.

  • Antonio Mendoza Pérez

    Temporal Technologies

    Staff Developer Success Engineer

  • Gaurav Thadani

    Temporal

    Staff Developer Success Engineer

Open session

AI Is the Easy Part: Why Faster Engineers Don't Make Faster Businesses

  • Bharat Sharma

    Booking

    Senior Engineering Manager

Open session

How is SWE & EM roles are changing in the age of AI

  • Amol Sharma

    Stripe

    Engineering Manager

Related videos

How building an industry DBMS differs from building a research one thumbnail
30:34
World Congress 2023 • about 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • 3 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Swapping Low Latency Data Storage Under High Load thumbnail
22:32
World Congress 2023 • about 3 years ago

Swapping Low Latency Data Storage Under High Load

George Asafev

Agentic employees in world's most downloaded FinTech app thumbnail
30:37
World Congress 2026 Europe • 3 months ago

Agentic employees in world's most downloaded FinTech app

Sasa Fajkovic

How to Destroy a Monolith? thumbnail
26:45
World Congress 2025 • about 1 year ago

How to Destroy a Monolith?

Babette Wagner

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.