WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2024 • Aug 20, 2024 • Session details

Modern Data Architectures need Software Engineering

Matthias Niehoff

Centralized data teams consistently emerge as workflow bottlenecks. Scaling modern architecture requires treating data as a product using Data Mesh, strict CI/CD contracts, and software engineering practices.

Pause
Mute Enter Fullscreen
#1 about 5 min

Evolution of centralized data architectures and open table formats

The transition from traditional data warehouses and lakes to modern cloud lakehouses simplifies scaling and data transformations.

#2 about 4 min

Decentralizing data bottlenecks with data mesh principles

Applying domain ownership and data-as-a-product concepts resolves the bottleneck issues created by centralized data teams.

#3 about 7 min

Applying software engineering environments and testing to data pipelines

Implementing unit testing and proper multi-environment deployment structures ensures robust data pipelines in production.

#4 about 5 min

Bringing DevOps practices to data transformation with DBT

Standardizing SQL transformations with infrastructure-as-code and automated testing enables robust continuous integration workflows.

#5 about 5 min

Bridging operational and analytical systems using formal data contracts

Treating data exports as formal agreements prevents upstream breaking changes and enforces programmatic accountability between teams.

#6 about 2 min

Cultivating data thinking and literacy across the organization

Training cross-functional staff in basic query paradigms reveals the business necessity of structuring data reliably.

#7 about 4 min

Modern tooling options and data team sizing constraints

Assembling architectures with lightweight analytical engines remains critical given the consistently smaller headcounts of data departments.

Matching moments

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · WWC Europe 2026

1:19 min

The true role and evolution of data engineering

Sandhya Menon Sandhya Menon · WWC Europe 2026

1:50 min

Introduction to the speaker and data science background

Bas Geerdink · LIVE

2:57 min

Core technical practices for robust data engineering

Sandhya Menon Sandhya Menon · WWC Europe 2026

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · WWC Europe 2026

1:50 min

The future of data engineering and AI mesh

Sandhya Menon Sandhya Menon · WWC Europe 2026

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
10 months ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
almost 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
almost 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail
24:08
almost 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail
27:46
almost 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Upcoming sessions on this topic

Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu
Open session

World Congress 2026 North America

Building the AI Era Through Developer Communities: A Competitive Advantage for Engineers & Teams

John Komarnicki

National Executive Advisor at Code & Coffee | Engineering & AI Community Leader

John Komarnicki
Open session

World Congress 2026 North America

It’s Alive! Taming the MLOps Franken-Stack: Write, Run, and Serve with Michelangelo

Paul Zimmerman, Eric Wang

Paul Zimmerman
Eric Wang

Related videos

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
WWC Europe 2026 • about 1 month ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

The Data Mesh as the end of the Datalake as we know it thumbnail
37:21
LIVE • about 5 years ago

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Enjoying SQL data pipelines with dbt thumbnail
26:53
WWC 2023 • almost 3 years ago

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Data Science on Software Data thumbnail
48:10
WWC 2021 • about 5 years ago

Data Science on Software Data

Markus Harrer

Building high performance and scalable architectures for enterprises thumbnail
58:23
LIVE • about 6 years ago

Building high performance and scalable architectures for enterprises

Luka Samaržija, Ivan Sokol

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail
28:42
WWC Europe 2026 • about 1 month ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.