Data Engineer

We are looking for a hands-on Data Engineer to build and maintain scalable Databricks pipelines combining demographic data with geographic reference datasets. You will develop reliable, reusable data processing solutions to support population analysis, demographic reporting and service planning, ensuring geographic assignments are accurate, traceable and reproducible., * Develop and maintain production data pipelines using Databricks, Python, PySpark and SQL.

• Ingest, transform and enrich demographic records using postcode directories, geographic boundaries and organisation reference data.
• Implement geospatial processing, including spatial joins, point-in-polygon matching, coordinate transformations and distance calculations.
• Manage Delta Lake datasets, incremental processing, historical records and versioned geographic reference data.
• Apply data quality checks, reconciliation and exception handling for missing postcodes, unmatched locations, duplicate records and invalid coordinates.
• Ensure accurate geographic assignments using appropriate reference-data versions and effective dates.
• Optimise Spark workloads for performance, scalability and efficient resource usage.
• Apply appropriate data access, pseudonymisation and disclosure controls when handling sensitive demographic and location data.
• Use Git, CI/CD, code reviews and automated testing to support controlled releases and reliable production operations.
• Work with analysts, data owners and platform teams to agree mapping rules, resolve data issues and document processing logic.

• Proven experience developing and supporting production-grade data pipelines using Delta Lake, including incremental ingestion, schema management, merge logic and deduplication.
• Practical experience in geospatial data engineering, including spatial joins, point-in-polygon matching, geometry validation and geographic aggregation.
• Understanding of coordinate reference systems, particularly British National Grid and WGS84, coordinate transformations and distance calculations.
• Experience with Spark-compatible spatial processing tools, such as Databricks spatial functions or Apache Sedona.
• Experience working with geospatial formats such as GeoJSON, WKT/WKB, Shapefile or GeoParquet.
• Strong data quality, reconciliation, testing and troubleshooting skills.
• Experience with Git, CI/CD, automated testing, job monitoring and performance optimisation.

Desirable Experience

• Healthcare demographic or patient registration data and UK postcode or organisation reference datasets.
• UK statistical and healthcare geographies, including Lower layer and Middle layer Super Output Areas (LSOAs and MSOAs), local authorities and regional healthcare planning areas.
• GeoPandas, Shapely, pyproj or QGIS.
• Address matching using Unique Property Reference Numbers (UPRNs) and approved address reference sources.
• Population, census or deprivation datasets, cloud object storage and experience handling sensitive demographic data in regulated environments.