Senior Data Engineer

• Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark, and Spark SQL following a full Medallion Architecture (Bronze - Silver - Gold).
• Implement ingestion patterns for files, APIs, SaaS platforms (e.g., subscription billing), SQL sources, SharePoint, and SFTP employing ADF and metadata-driven frameworks.
• Apply Lakeflow expectations for data quality, schema validation, and operational reliability.
• Build clean, conformed Silver/Gold models aligned to enterprise business domains such as customers, subscriptions, deliveries, finance, credit, logistics, and operations.
• Deliver star schemas, harmonization logic, SCDs, and business marts to enhance high-performance Power BI datasets.
• Apply governance, lineage, and fine-grained permissions via Unity Catalog.
• Design and optimize orchestration using Lakeflow Workflows and Azure Data Factory.
• Implement monitoring, alerting, SLAs/SLIs, runbooks, and cost optimization across the platform.
• Build CI/CD pipelines in Azure DevOps for notebooks, Lakeflow pipelines, SQL models, and ADF artifacts.
• Ensure secure, enterprise-grade platform operation utilizing private endpoints, managed identities, and Key Vault.
• Contribute to platform standards, design patterns, code reviews, and future roadmap.
• Collaborate closely with BI/Analytics teams to deliver curated datasets that support dashboards across the organization.
• Influence architecture decisions and elevate engineering maturity within our growing data function.
• ###

Technologies:

• Azure
• CI/CD
• Databricks
• DevOps
• Git
• Support
• Power BI
• PySpark
• SQL
• SharePoint
• Spark
• Unity
• Cloud
• GameDev
• Excel
• Fabric
• Python
• Security
• Terraform

More:

We are a leading UK consumer business currently on an exciting journey of data modernization, moving away from legacy systems and manual reporting. We are building a fully automated Azure Enterprise Landing Zone combined with Databricks Lakehouse, and we want your expertise to shape our new data platform. This is a rare opportunity for you to join early in the process, influence architecture, and define engineering standards that will support our Operations, Finance, Sales, Logistics, and Customer Care functions. If youre passionate about creating a best-in-class lakehouse from the ground up, we would love to hear from you!

• Significant commercial experience in Data Engineering, specifically with production workloads on Azure and Databricks.
• Strong proficiency in PySpark/Spark SQL and distributed data processing.
• Proven experience in delivering solutions based on Medallion/Lakehouse architecture using Delta Lake.
• Solid understanding of dimensional modeling (Kimball), including surrogate keys, SCD types 1 and 2, and merge strategies.
• Operational experience with SLAs, observability, idempotent pipelines, reprocessing, and backfills.
• Strong grounding in secure Azure Landing Zone patterns.
• Proficiency with Git, CI/CD, automated deployments, and modern engineering standards.
• Effective communication skills to translate technical decisions into business outcomes.
• ###