Senior Data Engineer/ Scientist

Head Resourcing Ltd

8 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Languages

English

Experience level

Senior

Compensation

£ 70K

Job location

Tech stack

Microsoft Excel

API

Data analysis

Azure

Software as a Service

Code Review

Continuous Integration

Information Engineering

Software Design Patterns

Distributed Data Store

Hive

SharePoint

SQL Databases

Data Streaming

File Transfer Protocol (FTP)

Azure

Spark

GIT

Data Lake

PySpark

Bicep

Terraform

Legacy Systems

Databricks

Job description

A rapidly scaling UK consumer brand is undertaking a major data modernisation programme-moving away from legacy systems, manual Excel reporting and fragmented data sources into a fully automated Azure Enterprise Landing Zone + Databricks Lakehouse. They are building a modern data platform from the ground up using Lakeflow Declarative Pipelines, Unity Catalog, and Azure Data Factory, and this role sits right at the heart of that transformation. This is a rare opportunity to join early, influence architecture, and help define engineering standards, pipelines, curated layers and best practices that will support Operations, Finance, Sales, Logistics and Customer Care. If you want to build a best-in-class Lakehouse from scratch-this is the one. ? What You'll Be DoingLakehouse Engineering (Azure + Databricks) * Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark, and Spark SQL across a full Medallion Architecture (Bronze ? Silver ? Gold). * Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing), SQL sources, SharePoint and SFTP using ADF + metadata-driven frameworks. * Apply Lakeflow expectations for data quality, schema validation and operational reliability., Build CI/CD pipelines in Azure DevOps for notebooks, Lakeflow pipelines, SQL models and ADF artefacts. * Ensure secure, enterprise-grade platform operation across Dev ? Prod, using private endpoints, managed identities and Key Vault. * Contribute to platform standards, design patterns, code reviews and future roadmap.

Collaboration & Delivery * Work closely with BI/Analytics teams to deliver curated datasets powering dashboards across the organisation. * Influence architecture decisions and uplift engineering maturity within a growing data function.

Requirements

5-8+ years of Data Engineering with 2-3+ years delivering production workloads on Azure + Databricks. * Strong PySpark/Spark SQL and distributed data processing expertise. * Proven Medallion/Lakehouse delivery experience using Delta Lake. * Solid dimensional modelling (Kimball) including surrogate keys, SCD types 1/2, and merge strategies. * Operational experience-SLAs, observability, idempotent pipelines, reprocessing, backfills.

Mindset * Strong grounding in secure Azure Landing Zone patterns. * Comfort with Git, CI/CD, automated deployments and modern engineering standards. * Clear communicator who can translate technical decisions into business outcomes., Streaming ingestion experience (Auto Loader, structured streaming, watermarking) * Subscription/entitlement modelling experience * Advanced Unity Catalog security (RLS, ABAC, PII governance) * Terraform/Bicep for IaC * Fabric Semantic Model / Direct Lake optimisation

Skills