Ferd Scheepers
A Data Mesh needs Open Metadata
#1about 5 minutes
The challenges of a centralized data lake architecture
ING's initial centralized data lake, which used a common format, struggled with manual ETL processes and scalability issues across a multi-cloud landscape.
#2about 6 minutes
Learning from the flower industry's data logistics
The evolution of the flower industry from a centralized to a decentralized logistics model illustrates how data can move directly while metadata remains coordinated.
#3about 4 minutes
Understanding the data mesh as a concept, not a product
The data mesh offers a powerful decentralized architecture, but its implementation is a significant engineering challenge as no single product exists.
#4about 3 minutes
Building an open source data mesh and metadata mesh
ING collaborates on open source projects like Fabric for the data mesh and identifies the need for a separate metadata mesh to solve interoperability issues.
#5about 5 minutes
Introducing Egeria as an open metadata standard
Egeria is an open source project under the Linux Foundation that provides a standard for exchanging metadata between different vendor tools via a two-layer API.
#6about 3 minutes
Managing an open source contribution team within a bank
ING's dedicated team contributes directly to the Egeria open source project, navigating the challenge of prioritizing a community-driven backlog over internal requests.
#7about 6 minutes
Using Egeria for unified search and regulatory compliance
ING leverages Egeria for unified metadata search and to automate end-to-end data lineage, which is crucial for meeting regulatory requirements like BCBS 239.
#8about 6 minutes
Q&A on data ownership, monetization, and competing tools
The Q&A session covers incentivizing data ownership through monetization, motivating metadata provision, and comparing Egeria's comprehensive model to other industry solutions.
Related jobs
Jobs that call for the skills explored in this talk.
Featured Partners
Related Videos
37:21The Data Mesh as the end of the Datalake as we know it
Mario Meir-Huber
29:25Modern Data Architectures need Software Engineering
Matthias Niehoff
32:12Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata
Clemens Vasters
20:04Data Governance in the Era of AI
Kateřina Ščavnická
28:29Unlocking Value from Data: The Key to Smarter Business Decisions-
Taqi Jaffri, Kapil Gupta & Farooq Sheikh and Tomislav Tipurić
19:44Blueprints for Success: Steering a Global Data & AI Architecture
Dominik Schneider
29:31Break the Chain: Decentralized solutions for today’s Web2.0 privacy problems
Adam Larter
21:17Data is Key: Scraping Metadata from Websites
Lars Kölker
From learning to earning
Jobs that call for the skills explored in this talk.
Senior Data Scientist - Conversation Analytics & Dashboarding (w/m/d)
ING Bank
Frankfurt am Main, Germany
Senior
Python
Plotly
Data analysis
Machine Learning
Data Scientist Tribe Mortgages
ING
Brussels, Belgium
Intermediate
Java
Azure
Data analysis
Machine Learning
ETL-/ELT-Entwickler - Data Warehouse (w/m/d) - Hybrid (Softwareentwickler/in)
ING Deutschland
€70-75K
ETL
Scrum
Python
Data Engineers - Talent Community | BASF Digital Hub Madrid
dark_blue_EMEA
Municipality of Madrid, Spain
Remote
ETL
Azure
Python