WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2024 • Aug 20, 2024 • Session details

Modern Data Architectures need Software Engineering

Matthias Niehoff

Centralized data teams consistently emerge as workflow bottlenecks. Scaling modern architecture requires treating data as a product using Data Mesh, strict CI/CD contracts, and software engineering practices.

Pause
Mute Enter Fullscreen
#1 about 5 min

Evolution of centralized data architectures and open table formats

The transition from traditional data warehouses and lakes to modern cloud lakehouses simplifies scaling and data transformations.

#2 about 4 min

Decentralizing data bottlenecks with data mesh principles

Applying domain ownership and data-as-a-product concepts resolves the bottleneck issues created by centralized data teams.

#3 about 7 min

Applying software engineering environments and testing to data pipelines

Implementing unit testing and proper multi-environment deployment structures ensures robust data pipelines in production.

#4 about 5 min

Bringing DevOps practices to data transformation with DBT

Standardizing SQL transformations with infrastructure-as-code and automated testing enables robust continuous integration workflows.

#5 about 5 min

Bridging operational and analytical systems using formal data contracts

Treating data exports as formal agreements prevents upstream breaking changes and enforces programmatic accountability between teams.

#6 about 2 min

Cultivating data thinking and literacy across the organization

Training cross-functional staff in basic query paradigms reveals the business necessity of structuring data reliably.

#7 about 4 min

Modern tooling options and data team sizing constraints

Assembling architectures with lightweight analytical engines remains critical given the consistently smaller headcounts of data departments.

Matching moments

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff Matthias Niehoff · World Congress 2026 Europe

1:19 min

The true role and evolution of data engineering

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

1:50 min

Introduction to the speaker and data science background

Bas Geerdink · LIVE

2:57 min

Core technical practices for robust data engineering

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

1:50 min

The future of data engineering and AI mesh

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
12 months ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
about 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
about 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail
24:08
about 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail
27:46
about 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Related videos

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

The Data Mesh as the end of the Datalake as we know it thumbnail
37:21
LIVE • over 5 years ago

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Enjoying SQL data pipelines with dbt thumbnail
26:53
World Congress 2023 • almost 3 years ago

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Data Science on Software Data thumbnail
48:10
World Congress 2021 • about 5 years ago

Data Science on Software Data

Markus Harrer

Building high performance and scalable architectures for enterprises thumbnail
58:23
LIVE • about 6 years ago

Building high performance and scalable architectures for enterprises

Luka Samaržija, Ivan Sokol

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail
28:42
World Congress 2026 Europe • 3 months ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.