World Congress 2024 • Aug 20, 2024 • Session details

Modern Data Architectures need Software Engineering

Matthias Niehoff

Centralized data teams consistently emerge as workflow bottlenecks. Scaling modern architecture requires treating data as a product using Data Mesh, strict CI/CD contracts, and software engineering practices.

Pause

Mute

Enter Fullscreen

#1 about 5 min

Evolution of centralized data architectures and open table formats

The transition from traditional data warehouses and lakes to modern cloud lakehouses simplifies scaling and data transformations.

#2 about 4 min

Decentralizing data bottlenecks with data mesh principles

Applying domain ownership and data-as-a-product concepts resolves the bottleneck issues created by centralized data teams.

#3 about 7 min

Applying software engineering environments and testing to data pipelines

Implementing unit testing and proper multi-environment deployment structures ensures robust data pipelines in production.

#4 about 5 min

Bringing DevOps practices to data transformation with DBT

Standardizing SQL transformations with infrastructure-as-code and automated testing enables robust continuous integration workflows.

#5 about 5 min

Bridging operational and analytical systems using formal data contracts

Treating data exports as formal agreements prevents upstream breaking changes and enforces programmatic accountability between teams.

#6 about 2 min

Cultivating data thinking and literacy across the organization

Training cross-functional staff in basic query paradigms reveals the business necessity of structuring data reliably.

#7 about 4 min

Modern tooling options and data team sizing constraints

Assembling architectures with lightweight analytical engines remains critical given the consistently smaller headcounts of data departments.

Matching moments

56 sec

Introduction to analytical data formats for software developers

Matthias Niehoff

Matthias Niehoff · World Congress 2026 Europe

1:19 min

The true role and evolution of data engineering

Sandhya Menon

Sandhya Menon · World Congress 2026 Europe

1:50 min

Introduction to the speaker and data science background

Bas Geerdink · LIVE

2:57 min

Core technical practices for robust data engineering

Sandhya Menon

Sandhya Menon · World Congress 2026 Europe

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon

Sandhya Menon · World Congress 2026 Europe

1:50 min

The future of data engineering and AI mesh

Sandhya Menon

Sandhya Menon · World Congress 2026 Europe

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail

almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail

about 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail

about 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail

about 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail

about 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Upcoming sessions on this topic

World Congress 2026 North America

September 24, 2026 · 16:50–17:20

Stage 9

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 2

Reinventing Testing Practices in the AI Era

Eric Deandrea

Java Champion & Senior Principal Software Engineer, IBM

Eric Deandrea

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 9

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan

World Congress 2026 North America

September 24, 2026 · 14:50–15:20

Stage 2

Databases in the Agent Era

Monica Sarbu

Founder and CEO of xata.io

Monica Sarbu

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 3

Real-Time Data Platforms at Trillion-Event Scale

Diptamay Sanyal

Principal Engineer | Data, AI & Cybersecurity Platforms

Diptamay Sanyal

World Congress 2026 North America

September 24, 2026 · 10:20–10:50

Tech Leaders Stage

The New Rules of Software Delivery

AJ Aitken, Moritz Plassnig

AJ Aitken

Moritz Plassnig

Related videos

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail

World Congress 2026 Europe • about 2 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

The Data Mesh as the end of the Datalake as we know it thumbnail

LIVE • over 5 years ago

The Data Mesh as the end of the Datalake as we know it

Mario Meir-Huber

Enjoying SQL data pipelines with dbt thumbnail

World Congress 2023 • almost 3 years ago

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Data Science on Software Data thumbnail

World Congress 2021 • about 5 years ago

Data Science on Software Data

Markus Harrer

Building high performance and scalable architectures for enterprises thumbnail

LIVE • about 6 years ago

Building high performance and scalable architectures for enterprises

Luka Samaržija, Ivan Sokol

Parquet, Delta, Iceberg & Ducklake - An introduction for developers thumbnail

World Congress 2026 Europe • about 2 months ago

Parquet, Delta, Iceberg & Ducklake - An introduction for developers

Matthias Niehoff

Related articles

AA

Ali Alp

Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)

Why Event-Driven Architecture Isn’t About Speed—And When You Actually Need It We’ve all seen it happen. An engineering team wants to feel like they belong in the “cool kids” club, so they grab Apac...

about 1 month ago

A developer workstation showing a code editor on one monitor, and a diagram on another

BB

Benedikt Bischof

Making Data Warehouses Fast: A Developer’s Story

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Adnan Rahic who teaches the audience how to make data warehouses.About the Speaker: Adnan i...

about 4 years ago

Making Data Warehouses Fast: A Developer’s Story

AB

Andre Braun, GitLab

Now is the time for industrialized software development

Now is the time for industrialized software development Recently, I received a letter from my car’s manufacturer alerting me to a recall. They had discovered a defective part and wanted to replace ...

about 1 year ago

A person developint HTML on a laptop with the sun shining on it.

BB

Benedikt Bischof

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Bas Geerdink who gave advice on MLOps.‍About the speaker:‍Bas is a programmer, scientist, a...

about 4 years ago

MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production

From learning to earning

Jobs that call for the skills explored in this talk.

Lead Software Engineer - Data Engineering

Barcelona, Spain

NoSQL DevOps Docker

Data Engineer (all genders welcome)

Wietmarschen, Germany

Remote Git Docker Power BI

11 days ago • Verified

Data Governance Lead

Boston, United States

Expert Remote PostgreSQL Data analysis Data Warehouse

about 2 months ago • Verified

Cloud Data Engineer (all genders)

Hamburg, Germany

Experienced Remote Python DevOps Cloud (AWS/Google/Azure)

Data Engineer Information Technology

Expert €60k Git JIRA Dask

11 days ago • Verified

Lead Data Product Owner - Finance

Boston, United States

Expert Remote JIRA PostgreSQL Data analysis