WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 10, 2026 • Session details

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

A central data warehouse often transforms into a single point of failure. Learn how adopting a hybrid Data Mesh empowers domain teams and eliminates engineering bottlenecks.

Pause
Mute Enter Fullscreen
#1 about 2 min

Resolving the problem of messy ad-hoc queries

Unstructured SQL queries and scattered data files create silos and erode organizational trust.

#2 about 2 min

The true role and evolution of data engineering

Modern data engineering moves beyond query optimization to focus on reliable data movement, scalability, and automation.

#3 about 3 min

Transforming data architecture from on-premise to cloud

Transitioning from an on-premise database to Snowflake cloud storage utilizes Data Vault modeling to handle complex business acquisitions.

#4 about 2 min

The bottleneck of centralized data team ownership

Centralized data teams struggle to keep up with increasing business demands, leading to delayed innovations and employee frustration.

#5 about 3 min

Adopting a decentralized data mesh architecture model

Implementing data mesh principles distributes ownership to domain teams while maintaining federated governance and cross-domain data sharing.

#6 about 3 min

Empowering domain teams with an open data platform

Rethinking the central team's mandate to supply decentralized infrastructure and tools like Snowflake, dbt, and Airflow.

#7 about 3 min

Refactoring complex logic into scalable data products

Transforming a rigid top customer SQL query into a reusable, automated data pipeline establishes monitoring and organization-wide consensus.

#8 about 3 min

Core technical practices for robust data engineering

Essential disciplines include data modeling, pipeline orchestration, Git version control, standardization, and proactive monitoring.

#9 about 2 min

Common pitfalls in scaling modern data systems

Avoiding over-engineering, prioritizing documentation, securing stakeholder alignment, and planning for ongoing maintenance ensure long-term platform health.

#10 about 2 min

The future of data engineering and AI mesh

Preparing for metadata-driven platforms and AI mesh requires cementing a solid, scalable data foundation first.

#11 about 7 min

Strategies for transferring data ownership and adapting tools

Migrating responsibilities to domain teams demands continuous learning and adapting data tooling amidst rapid AI advancements.

Matching moments

1:50 min

Introduction to the speaker and data science background

Bas Geerdink · LIVE

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

1:54 min

Embedding data engineering to solve database scalability

Mikael Robert Mikael Robert · World Congress 2025

3:11 min

Decentralizing data bottlenecks with data mesh principles

Matthias Niehoff Matthias Niehoff · World Congress 2024

2:14 min

Solving complex platform architecture challenges at an enterprise scale

Maria Apazoglou · Coffee With Developers

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 2 months ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 2 months ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 2 months ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 2 months ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 2 months ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 2 months ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 3

Culture Doesn't Scale Itself: Leading Engineering Teams Through Hypergrowth and the AI Transition

Thanos Baskous

VP of Engineering and Co-founder of Cogent Security

Thanos Baskous
Open session

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 9

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal
Open session

World Congress 2026 North America

September 24, 2026 · 11:10–11:15

Outdoor Stage

Architecting the 100X SDLC: Building Production Trust into AI-Assisted Delivery

Ranjan Parthasarathy

Founder, CPTO/CEO at AXIOMSTUDIO.AI

Ranjan Parthasarathy
Open session

World Congress 2026 North America

September 25, 2026 · 16:10–16:40

Stage 1

The Five Percent Club: The Culture and Technological Shift Behind Successful AI Deployments

Tara Hernandez

Tara Hernandez, VP of Developer Productivity at MongoDB

Tara Hernandez
Open session

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
World Congress 2024 • about 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

Data: The Deciding Factor in AI Success thumbnail
28:53
World Congress 2026 Europe • about 2 months ago

Data: The Deciding Factor in AI Success

Damandeep Kochhar, Jörg Tewes, Siddhika Nevrekar, Ed Huang, Laura Moritz

The Data Mesh as the end of the Datalake as we know it thumbnail
37:21
LIVE • over 5 years ago

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Data Governance in the Era of AI thumbnail
20:04
World Congress 2025 • about 1 year ago

Data Governance in the Era of AI

Kateřina Ščavnická

Blueprints for Success: Steering a Global Data & AI Architecture thumbnail
19:44
World Congress 2025 • about 1 year ago

Blueprints for Success: Steering a Global Data & AI Architecture

Dominik Schneider

From Data Mesh to AI Mesh: Integrating Distributed Intelligence on Decentralized Data Architectures thumbnail
45:31
Europe 2026 Virtual • 2 months ago

From Data Mesh to AI Mesh: Integrating Distributed Intelligence on Decentralized Data Architectures

Olga Woschitz, Thomas Görz

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.