Data Engineer

An experienced data engineer with strong Azure Databricks expertise is required for a 6 month contract to develop and deliver a scalable data pipeline that ingests files and transforms them into standardised, high-quality datasets for reporting. This is a hands-on delivery role focused on building a production-grade pipeline, suited to someone comfortable working independently, defining technical solutions, and rapidly delivering high-quality data engineering components., * Implement a scalable ETL pipeline in Azure Databricks, that transforms raw data into standardised schemas to support reporting and analytics, following a medallion architecture in line with internal design standards.

• Leverage an existing email ingestion tool to extract multiple format source files from a shared mailbox.
• Develop notebooks, jobs and pipelines in Databricks for ETL orchestration.
• Design and implement robust validation rules, monitor pipeline runs and troubleshoot failures, with logging and notifications to support robust data processes.
• Integrate with CI/CD flows across environments using Azure DevOps.
• Produce clear technical documentation of the solution to ensure maintainability post-delivery.

Do you have experience in Unity?, * 2+ years hands-on experience with Azure Databricks, including data validation, transformation, and optimization.

• Expertise in PySpark/Spark SQL.
• Knowledge of Unity Catalog and modern Azure Databricks features, as well as data best practices and concepts.
• Proficiency building end-to-end data pipelines with Delta Lake/Lakehouse architecture.
• Experience implementing CI/CD pipelines (Azure DevOps preferred).
• Experience supporting BI reporting layers (Power BI or similar) is nice to have.
• Insurance experience is nice to have.