> Markdown version of [/jobs/ext/1494238-lead-data-engineer-cross-domain](https://www.wearedevelopers.com/jobs/ext/1494238-lead-data-engineer-cross-domain). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer (Cross Domain) - **Company:** DSM - **Location:** Barcelona, Spain - **Contract:** Permanent contract - **Skills:** Automation of Tests, Microsoft Azure, Business Intelligence Development, Cloud Database, Code Review, Continuous Integration, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Mart, Data Systems, Data Vault Modeling, DevOps, Github, Python (Programming Language), Azure DevOps Pipelines, Software Configuration Management, SQL Databases, Enterprise Data Management, Data Processing, Data Ingestion, Delivery Pipeline, Apache Spark, Change Data Capture, Git, Pyspark, Git Flow, Infrastructure Automation Frameworks, Deployment Automation, Performance Monitor, Bicep, Code Inspection, Data Management, Api Design, Terraform, Data Pipelines, Databricks - **Published:** July 30, 2026 - **Apply:** https://dejobs.org/x/x/78F2EAF7149447499169836F45062D78/job/ ## About the Role * Strong technical expertise in dbt, SQL, Python, Spark (PySpark), Databricks, Git, Azure DevOps, and GitHub, with proven experience designing, building, and maintaining production-grade data pipelines for 5+ years. * Deep CI/CD and automation experience, including the design and implementation of deployment pipelines using Azure DevOps Pipelines and GitHub Actions, with Infrastructure-as-Code knowledge (Terraform, Bicep) considered a strong advantage. * Advanced Git and repository management skills, including branching strategies, pull request standards, code review best practices, and governance across Azure DevOps and GitHub environments. * Strong cloud data engineering expertise (Azure preferred), with hands-on experience deploying, managing, and operating Databricks jobs, clusters, and workflows in production environments. * Extensive knowledge of Data Vault 2.0 architecture, including Raw Vault (Hubs, Links, Satellites), Business Vault components (bSAT, PIT, Bridge), and Gold-layer data marts and semantic models. * Expertise in data ingestion and integration patterns, including batch ETL, incremental processing, Change Data Capture (CDC), and API-based ingestion from a wide range of source systems. * Strong focus on data governance and quality, with experience applying FAIR data principles, implementing data quality automation, and managing enterprise data platforms; exposure to scientific datasets such as cheminformatics, bioinformatics, or microbiome data is an added advantage. * Leadership-oriented and collaborative mindset, combining curiosity, continuous improvement, accountability, and end-to-end ownership with the ability to lead teams, oversee projects, engage stakeholders, and work effectively across functions, geographies, and diverse teams. ## Description Join us as a Lead Data Engineer to design and maintain robust data pipelines, drive the best deployment practices, and manage key initiatives in the Procurement domain. You will own the full lifecycle of data products - from source ingestion to Gold layer consumption - using modern DevOps practices across Azure DevOps and GitHub. You will collaborate with data modelers, data stewards, and business partners to deliver impactful, production-grade data solutions. Elevate your career by mastering cutting-edge tools - dbt, Databricks, PySpark, and CI/CD automation - in a dynamic, innovative, and global environment. Your key responsibilities * Design and implement end-to-end data pipelines for ingestion, transformation, and storage across Bronze (raw), Silver (Raw Vault and Business Vault), and Gold (data marts and semantic layer), ensuring scalable and reliable data processing. * Develop and manage data ingestion processes from source systems through ETL and API-based extraction methods where platform tables are unavailable, ensuring seamless integration of enterprise data sources. * Build modular, reusable, and maintainable transformation frameworks using dbt, PySpark, and SQL, including the creation of staging models (raw_stage and stage) with hash keys, hash diffs, business keys, and load metadata in alignment with Data Vault 2.0 and modern engineering best practices. * Design, build, and maintain CI/CD and DevOps processes in Azure DevOps and GitHub Actions, incorporating automated testing, linting, deployment gates, environment promotion, and standardized branching, pull request, code review, and merge strategies. * Implement and manage Infrastructure-as-Code (IaC) and deployment automation for data platform resources, including provisioning environments, deploying dbt jobs and Databricks workflows, configuring clusters, managing dependencies, scheduling jobs, and maintaining environment-specific configurations. * Establish proactive monitoring, observability, and operational excellence practices by implementing alerting, conducting regular pipeline maintenance and upgrades, and troubleshooting issues to ensure platform reliability, performance, and rapid incident resolution. * Drive data quality, governance, and compliance standards through automated dbt testing, validation frameworks, FAIR data principles, lineage management, auditability, and data discoverability across all layers of the platform. * Provide technical leadership and cross-functional collaboration by evaluating solution feasibility, leading workload planning, mentoring data engineering teams, defining engineering standards, and partnering with data engineers, modelers, stewards, BI developers, data scientists, and business SMEs to deliver impactful data solutions. We offer * Unique career paths across health, nutrition, and beauty - explore what drives you and get the support to make it happen. * A chance to impact millions of consumers every day - sustainability embedded in all we do. * A science-led company with cutting-edge research and creativity everywhere - from biotech breakthroughs to sustainability game-changers, you'll work on what's next. * Growth that keeps up with you - you join an industry leader that will develop your expertise and leadership. * A culture that lifts you up - with collaborative teams, shared wins, and people who cheer each other on. * A community where your voice matters - your ideas are essential to our future. ## Related Videos - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Back(end) to the Future: Embracing the continuous Evolution of Infrastructure and Code](https://www.wearedevelopers.com/videos/440-back-end-to-the-future-embracing-the-continuous-evolution-of-infrastructure-and-code) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)