> Markdown version of [/videos/505-a-data-mesh-needs-open-metadata?t=1722](https://www.wearedevelopers.com/videos/505-a-data-mesh-needs-open-metadata?t=1722). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# A Data Mesh needs Open Metadata

Decentralized data meshes fail without standardized interoperability. Discover how the open-source metadata standard Egeria automates end-to-end governance and eliminates integration bottlenecks across multi-cloud environments.

- **Speakers:** Ferd Scheepers
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 38:36
- **URL:** https://www.wearedevelopers.com/videos/505-a-data-mesh-needs-open-metadata

## Summary

Organizations migrating toward a data mesh architecture face a massive logistical bottleneck: routing decentralized data between producers and consumers while maintaining governance and semantic consistency. While physical logistics (like the flower industry) or simple files (like JPEGs) inherently bundle payloads with their metadata, enterprise platforms traditionally strip this context. Relying on highly manual ETL processes or centralizing operations into a monolithic data lake fails to scale, leaving data engineering teams overwhelmed by custom integrations across multi-cloud environments.

To solve the interoperability challenge of a true data mesh, ING co-created Egeria, an open-source metadata standard under the Linux Foundation. Instead of waiting for a mythical out-of-the-box data mesh product to hit the market, Egeria standardizes metadata exchange through common APIs, enabling cross-platform integration between disparate tools like IBM Watson Knowledge Catalog, Microsoft Purview, and custom repositories. This ensures that as data moves across Apache Kafka queues, SQL databases, or APIs, its semantic meaning, quality metrics, and governance rules travel inherently with it.

Implementing an overarching metadata mesh unlocks vital operational capabilities. It enables federated search to locate consistent business definitions—such as pinpointing "customer" data across an entire technical footprint. Furthermore, it automates end-to-end data lineage visualization, replacing quickly outdated manual documentation with real-time reporting suitable for strict regulatory requirements like BCBS 239 in banking. Ultimately, treating data as a primary, monetizable "data product" rather than a byproduct drives the cultural shift necessary to sustain decentralized data ownership and governance.

**Keywords:** data mesh architecture, open metadata standards, egeria open source, data lineage visualization, federated metadata search, decentralized data products, multi-cloud data routing, ETL pipeline bottlenecks, end-to-end data governance, cross-platform interoperability, semantic data modeling, BCBS 239 regulatory compliance, apache kafka metadata integration, API data tracing, enterprise data monetization

## Chapters

1. **Embracing open source engineering in a digital bank** (00:05) — Building internal engineering talent to actively contribute to open source software establishes technical leadership.
1. **Standardizing data consumption with a common language** (01:37) — Defining shared business terms through a canonical format simplifies data movement between producers and consumers.
1. **Challenges of manual data mapping and localization** (03:50) — Moving and transforming data across global private and public clouds remains a highly manual logistical hurdle.
1. **Applying flower auction logistics to data metadata** (05:19) — Using metadata as a physical label enables highly automated transportation and logistics for moving diverse items.
1. **Bundling data and metadata for seamless integration** (08:11) — Shipping technical descriptions directly alongside assets eliminates friction during platform exchanges.
1. **Introducing data mesh and localized autonomy** (11:00) — Breaking down centralized dependencies empowers local groups to construct autonomous data products while maintaining global governance.
1. **Navigating the lack of mature data mesh tooling** (12:50) — Implementing localized data autonomy currently requires massive engineering effort to wire together disparate vendor products.
1. **Building the open source fabric data mesh product** (15:39) — Collaborating across organizations to build automated policies and data movement guardrails accelerates data mesh adoption.
1. **Bridging the semantic gap with interoperable metadata** (16:57) — Connecting different platforms requires reliable APIs and formats that preserve context and data quality.
1. **Creating standard APIs via the Egeria open metadata project** (18:44) — Two layers of integration and consumption APIs allow disparate catalogs and repositories to sync structural definitions seamlessly.
1. **Applying the Egeria metadata mesh for federated catalogs** (20:52) — Connecting multiple vendor setups into a federated cohort enables single-pane visibility across independent data environments.
1. **Fostering open source contributions in corporate engineering teams** (23:30) — Allowing engineers to prioritize community-driven open source features over traditional internal business requests fosters innovation.
1. **Enabling cross-platform metadata search and discovery** (26:54) — Indexing business terms, assets, and streaming topics allows technical teams to instantly locate enterprise-wide definitions.
1. **Mapping end-to-end data lineage for regulatory compliance** (28:42) — Rendering reality-based flow graphs proves compliance to auditors without relying on outdated manual documentation.
1. **Advancing industry standards by open sourcing core capabilities** (30:35) — Sharing intellectual property publicly encourages vendor adoption and prevents proprietary lock-in for critical infrastructure.
1. **Shifting culture toward monetizing data as a product** (32:54) — Treating data pipelines as primary revenue drivers incentivizes upstream teams to claim ownership and ensure quality.
1. **Incentivizing domains to produce rich data descriptions** (35:16) — Applying regulatory mandates forces baseline documentation, but rewarding domains for dataset creation drives sustainable metadata quality.
1. **Comparing Egeria to alternative open metadata solutions** (36:42) — A richer core type system provides deeper technical integration capabilities compared to visually appealing but shallow alternatives.

## Related Moments

- [The future of data engineering and AI mesh](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) (from "From Messy Queries to Scalable Systems - How Data Engineering actually works")
- [Decentralizing data bottlenecks with data mesh principles](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) (from "Modern Data Architectures need Software Engineering")
- [Adopting a decentralized data mesh architecture model](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) (from "From Messy Queries to Scalable Systems - How Data Engineering actually works")
- [Understanding data mesh as an organizational design shift](https://www.wearedevelopers.com/videos/156-the-data-mesh-as-the-end-of-the-datalake-as-we-know-it) (from "The Data Mesh as the end of the Datalake as we know it")
- [Adopting data mesh ownership and aggregated data models](https://www.wearedevelopers.com/videos/1622-data-governance-in-the-era-of-ai) (from "Data Governance in the Era of AI")
- [Fostering community innovation through transparent data interoperability and ownership](https://www.wearedevelopers.com/videos/635-web3-embracing-the-next-era-of-the-internet-within-a-web2-landscape) (from "Web3: Embracing the Next Era of the Internet Within a Web2 Landscape")

## Related Articles

- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [WebMCP: Empowering Agents as First-Class Citizens of the Web](https://www.wearedevelopers.com/magazine/696-webmcp-empowering-agents-as-first-class-citizens-of-the-web)
- [The Future of Open Source: A Deep Dive - Scott Chacon at WeAreDevelopers World Congress 2024](https://www.wearedevelopers.com/magazine/471-the-future-of-open-source-a-deep-dive-scott-chacon-at-wearedevelopers-world-congress-2024)

## Related Jobs

- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [MDM Data Specialist (L2)](https://www.wearedevelopers.com/jobs/ext/1133506-mdm-data-specialist-l2) at **Twilio**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Platform Engineer - Mercury Runtime Platform](https://www.wearedevelopers.com/jobs/ext/293235-platform-engineer-mercury-runtime-platform) at **Raiffeisen Bank International AG**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Cloud Data Engineer (all genders)](https://www.wearedevelopers.com/jobs/48216-cloud-data-engineer-all-genders) at **msg**