Data Engineer

• Design, build and support data pipelines using Databricks, Python, PySpark and SQL
• Develop reusable ingestion, transformation, validation and enrichment processes
• Ingest demographic datasets, postcode data, geographic boundaries and reference datasets
• Build and maintain Delta Lake datasets and manage data lineage
• Perform postcode enrichment and geographic assignment processing
• Implement spatial joins, point-in-polygon matching and location-based calculations
• Manage geographic reference data versions and historical reproducibility
• Develop automated data quality controls and reconciliation processes
• Optimise Spark workloads and pipeline performance
• Support production releases, monitoring and troubleshooting activities
• Work closely with analysts, data owners and platform teams to define business rules and acceptance criteria

• Strong commercial experience with:
• Databricks
• Python
• PySpark
• SQL
• Proven experience building and supporting production-grade data pipelines
• Strong knowledge of Delta Lake, schema management, merge logic and incremental processing
• Experience handling large datasets within cloud-based environments
• Practical experience working with geospatial data and spatial processing
• Understanding of coordinate reference systems, including:
• British National Grid (BNG)
• WGS84
• Experience with spatial joins, geometry validation and geographic aggregation
• Strong data quality, reconciliation and testing experience
• Experience with Git, CI/CD and automated deployment pipelines

Desirable Experience

• Apache Sedona or Databricks Spatial Functions
• GeoPandas, Shapely, pyproj or QGIS
• NHS or healthcare data
• NHS Postcode Directory (NHSPD)
• ONS Postcode Directory (ONSPD)
• National Statistics Postcode Lookup (NSPL)
• Experience working with:
• LSOAs
• MSOAs
• Local Authorities
• Integrated Care Boards (ICBs)
• GeoJSON, GeoParquet, Shapefiles, WKT/WKB formats
• Population, census or demographic datasets
• Experience working within regulated or secure environments

What We’re Looking For

The ideal candidate will have a strong Data Engineering background combined with experience working with geographic and location-based datasets. You will be comfortable building scalable Databricks solutions, implementing automated processes, and ensuring the accuracy and reliability of demographic and geospatial data.

Candidates must be eligible to obtain SC Clearance, with active clearance highly advantageous. What’s on OfferContract

• Up to £550 per day
• Inside IR35
• Predominantly remote working

Permanent

• Up to £90,000 total package
• Base salary up to approximately £81,000
• Performance bonus
• Predominantly remote working