WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2026 Europe • Jul 10, 2026 • Session details

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

A central data warehouse often transforms into a single point of failure. Learn how adopting a hybrid Data Mesh empowers domain teams and eliminates engineering bottlenecks.

Pause
Mute Enter Fullscreen
#1 about 2 min

Resolving the problem of messy ad-hoc queries

Unstructured SQL queries and scattered data files create silos and erode organizational trust.

#2 about 2 min

The true role and evolution of data engineering

Modern data engineering moves beyond query optimization to focus on reliable data movement, scalability, and automation.

#3 about 3 min

Transforming data architecture from on-premise to cloud

Transitioning from an on-premise database to Snowflake cloud storage utilizes Data Vault modeling to handle complex business acquisitions.

#4 about 2 min

The bottleneck of centralized data team ownership

Centralized data teams struggle to keep up with increasing business demands, leading to delayed innovations and employee frustration.

#5 about 3 min

Adopting a decentralized data mesh architecture model

Implementing data mesh principles distributes ownership to domain teams while maintaining federated governance and cross-domain data sharing.

#6 about 3 min

Empowering domain teams with an open data platform

Rethinking the central team's mandate to supply decentralized infrastructure and tools like Snowflake, dbt, and Airflow.

#7 about 3 min

Refactoring complex logic into scalable data products

Transforming a rigid top customer SQL query into a reusable, automated data pipeline establishes monitoring and organization-wide consensus.

#8 about 3 min

Core technical practices for robust data engineering

Essential disciplines include data modeling, pipeline orchestration, Git version control, standardization, and proactive monitoring.

#9 about 2 min

Common pitfalls in scaling modern data systems

Avoiding over-engineering, prioritizing documentation, securing stakeholder alignment, and planning for ongoing maintenance ensure long-term platform health.

#10 about 2 min

The future of data engineering and AI mesh

Preparing for metadata-driven platforms and AI mesh requires cementing a solid, scalable data foundation first.

#11 about 7 min

Strategies for transferring data ownership and adapting tools

Migrating responsibilities to domain teams demands continuous learning and adapting data tooling amidst rapid AI advancements.

Matching moments

1:50 min

Introduction to the speaker and data science background

Bas Geerdink · LIVE

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · WWC Europe 2026

1:54 min

Embedding data engineering to solve database scalability

Mikael Robert Mikael Robert · WWC 2025

3:11 min

Decentralizing data bottlenecks with data mesh principles

Matthias Niehoff Matthias Niehoff · WWC 2024

2:14 min

Solving complex platform architecture challenges at an enterprise scale

Maria Apazoglou · Coffee With Developers

More from World Congress 2026 Europe

Building the Nervous System of AI - Michael Kagan (NVIDIA) thumbnail
29:17
about 1 month ago

Building the Nervous System of AI - Michael Kagan (NVIDIA)

Michael Kagan, Sead Ahmetović

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote thumbnail
19:41
about 1 month ago

Five Things in Tech that Matter Now - WeAreDevelopers World Congress 2026 Closing Keynote

Christian Heilmann

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend thumbnail
32:06
about 1 month ago

The Sound of Your Secrets: Teaching Your Model to Spy, So You Can Learn to Defend

David vonThenen

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes thumbnail
35:13
about 1 month ago

Super scaling for the Super Bowl: How to survive 30 million users hitting your backend in 30 minutes

Irina Branovic

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬 thumbnail
24:23
about 1 month ago

Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬

Jeremy Murray

Angstfreude - AI and Corporate Culture - The Thrill and the Threat thumbnail
30:26
about 1 month ago

Angstfreude - AI and Corporate Culture - The Thrill and the Threat

Axel Ebert, Karin Krobath

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Culture Doesn't Scale Itself: Leading Engineering Teams Through Hypergrowth and the AI Transition

Thanos Baskous

VP of Engineering and Co-founder of Cogent Security

Thanos Baskous
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

Making Science Larger, not just Faster

Yuval Dvir

Commercial Executive, SandboxAQ

Yuval Dvir
Open session

World Congress 2026 North America

Evals Are Infra: Building AI Systems Developers Can Actually Trust

Phoebe Wang

Member of Technical Staff at OpenAI

Phoebe Wang

Related videos

Modern Data Architectures need Software Engineering thumbnail
29:25
WWC 2024 • almost 2 years ago

Modern Data Architectures need Software Engineering

Matthias Niehoff

Data: The Deciding Factor in AI Success thumbnail
28:53
WWC Europe 2026 • about 1 month ago

Data: The Deciding Factor in AI Success

Damandeep Kochhar, Jörg Tewes, Siddhika Nevrekar, Ed Huang, Laura Moritz

The Data Mesh as the end of the Datalake as we know it thumbnail
37:21
LIVE • about 5 years ago

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Data Governance in the Era of AI thumbnail
20:04
WWC 2025 • 12 months ago

Data Governance in the Era of AI

Kateřina Ščavnická

Blueprints for Success: Steering a Global Data & AI Architecture thumbnail
19:44
WWC 2025 • 12 months ago

Blueprints for Success: Steering a Global Data & AI Architecture

Dominik Schneider

How building an industry DBMS differs from building a research one thumbnail
30:34
WWC 2023 • almost 3 years ago

How building an industry DBMS differs from building a research one

Markus Dreseler

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.