Data Engineer

You will work with data owners, analysts and platform teams to translate requirements into reusable processing. The role includes postcode enrichment, spatial processing, data quality and management of changing geographic reference data, from development through release and production support., * Design, develop and deploy data pipelines using Databricks, Python, PySpark and SQL. Build reusable ingestion, transformation and validation components, with reliable scheduling, incremental processing and safe reprocessing.

• Ingest demographic records, postcode directories, geographic boundaries and organisation reference data. Maintain source metadata, data lineage and versioned raw and curated datasets using Delta Lake.
• Standardise postcode and address fields while preserving source values. Apply agreed record keys and effective dates, and manage invalid, missing, terminated or unmatched postcodes through documented exception rules.
• Enrich records using approved NHS and ONS postcode products and lookup tables. Derive agreed statistical, local authority and health geographies, and validate organisation and site codes against ODS reference data.
• Implement spatial joins, including point-in-polygon matching, and distance calculations where required. Validate geometry and coordinate reference systems, and define how boundary-edge cases and multiple matches are handled.
• Maintain versions and effective dates for postcode lookups, boundaries and organisation changes. Apply the correct geographic vintage for each reporting period so historical outputs remain reproducible.
• Keep residential geography distinct from registered-practice and provider geography. Agree demographic definitions, age reference dates and aggregation rules with analysts to avoid incorrect linkage or double counting.
• Automate checks for completeness, duplicate records, unmatched locations, invalid coordinates and unexpected geography changes. Reconcile counts before and after joins and report exceptions for investigation.
• Optimise Spark and spatial processing through appropriate partitioning, join strategies and spatial indexing where supported. Monitor runtime, data volumes and resource use as pipelines scale.
• Apply approved access, pseudonymisation and disclosure controls to demographic and location data. Use authorised reference and geocoding services, and produce outputs at the agreed geographic granularity.
• Use code review, automated tests and controlled releases. Document mappings, reference-data versions and recovery procedures, and investigate production issues with data owners and support teams.
• Strong Databricks, Python, PySpark and SQL skills, with hands-on experience developing and supporting production data pipelines processing large datasets.

• Experience with Delta Lake, batch and incremental ingestion, schema management, merge logic, deduplication and maintaining historical records without losing the meaning of effective dates.
• Practical geospatial data engineering experience working with points, polygons, spatial joins, geometry validation and geographic aggregation.
• Understanding of coordinate reference systems, including British National Grid and WGS84, coordinate transformation, distance units and the limitations of postcode centroids for precise location analysis.
• Experience with a scalable Spark spatial approach, such as supported native Databricks spatial functions or Apache Sedona. Able to select tooling compatible with the deployed runtime and workload.
• Ability to ingest and transform spatial formats such as GeoJSON, WKT/WKB and Shapefile or GeoParquet, alongside tabular reference files.
• Experience joining demographic or other record-level data to geographic reference data, handling unmatched and multiple matches, and controlling changes to lookup versions and boundaries.
• Strong data quality and reconciliation skills, including tests that identify record multiplication, missing mappings and incorrect geographic assignments.
• Experience with Git, CI/CD, code review, automated testing, job monitoring and performance troubleshooting. Able to document assumptions and explain data issues clearly to analysts and data owners.