Senior Data Engineer(Databricks)

Senior Data Engineer(Databricks) Location: Remote from Spain (an indefinite Spanish employment contract) We are hiring a Senior Data Engineer to join an Intellias delivery team on a large-scale enterprise data platform migration programme for a financial services client.This is a hands-on senior role.You will write production code every week, own architectural decisions on your workstream, and mentor a paired mid-level engineer.Project Overview: Our client is an independent, active global asset manager with over R3 trillion in assets under management.They are executing a firm-wide data strategy to govern, manage and engineer data as a product across more than 70 business-owned sub-domains.As part of this, they are migrating to Databricks as the foundational layer of their Enterprise Data Platform, adopting a lakehouse architecture built on open formats, declarative pipelines and Unity Catalog.Requirements: - 6+ years of data engineering experience, with at least the last three years on Databricks in production.- Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.- Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.- Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.- Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.- CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).- Fluent working English.Nice to have - Active Databricks certifications - Prior exposure to investment management platforms, asset management operations data, market data feeds from major providers - CDMP DAMA certification or equivalent data governance credential.- Experience applying agentic engineering tooling to production data engineering (not just personal productivity), including MCP-connected Databricks or cloud servers.Responsibilities: - Perform the end-to-end delivery of one or more priority data domains on Azure Databricks.- Design and build medallion (Bronze / Silver / Gold) pipelines using Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming and Delta Lake with Liquid Clustering.- Register curated data products in Unity Catalog with the correct tags, masks, row filters, lineage and access policies.- Implement data quality gates at the Silver-to-Gold boundary, and refine rules with domain stewards.- Package and deploy work using Databricks Asset Bundles through Azure DevOps CI/CD - Write automated tests and documentation - Work at AI-as-Collaborator level today with a path to AI-as-Orchestrator, using Claude Code, Databricks Assistant, GitHub Copilot and MCP-connected agent tooling to accelerate the recurring parts of pipeline build, test and deploy.Why this position: Build within a large-scale enterprise data platform migration from day one, designing and delivering the pipelines, ingestion patterns and domain migrations that bring a 70+ sub-domain data strategy to life.You’ll work hands-on with the leading edge of the Databricks ecosystem on a high-stakes lakehouse build, applying agentic engineering tooling to accelerate delivery, and working closely with the architecture team to turn platform standards into working, production-grade data.

Requirements: - 6+ years of data engineering experience, with at least the last three years on Databricks in production.

• Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.
• Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.
• Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.
• Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.
• CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).

• Fluent working English. Nice to have - Active Databricks certifications - Prior exposure to investment management platforms, asset management operations data, market data feeds from major providers - CDMP DAMA certification or equivalent data governance credential.

• Experience applying agentic engineering tooling to production data engineering (not just personal productivity), including MCP-connected Databricks or cloud servers.