A Data Mesh needs Open Metadata

Your data mesh is incomplete. It needs a metadata mesh to enable federated search and automated lineage for compliance.

#1about 5 minutes

The challenges of a centralized data lake architecture

ING's initial centralized data lake, which used a common format, struggled with manual ETL processes and scalability issues across a multi-cloud landscape.

#2about 6 minutes

Learning from the flower industry's data logistics

The evolution of the flower industry from a centralized to a decentralized logistics model illustrates how data can move directly while metadata remains coordinated.

#3about 4 minutes

Understanding the data mesh as a concept, not a product

The data mesh offers a powerful decentralized architecture, but its implementation is a significant engineering challenge as no single product exists.

#4about 3 minutes

Building an open source data mesh and metadata mesh

ING collaborates on open source projects like Fabric for the data mesh and identifies the need for a separate metadata mesh to solve interoperability issues.

#5about 5 minutes

Introducing Egeria as an open metadata standard

Egeria is an open source project under the Linux Foundation that provides a standard for exchanging metadata between different vendor tools via a two-layer API.

#6about 3 minutes

Managing an open source contribution team within a bank

ING's dedicated team contributes directly to the Egeria open source project, navigating the challenge of prioritizing a community-driven backlog over internal requests.

#7about 6 minutes

Using Egeria for unified search and regulatory compliance

ING leverages Egeria for unified metadata search and to automate end-to-end data lineage, which is crucial for meeting regulatory requirements like BCBS 239.

#8about 6 minutes

Q&A on data ownership, monetization, and competing tools

The Q&A session covers incentivizing data ownership through monetization, motivating metadata provision, and comparing Egeria's comprehensive model to other industry solutions.