Senior Data Engineer

The successful candidate will work closely with stakeholders to improve reporting, resolve data issues and support better decision-making across the organisation., * Design, build and maintain data pipelines from APIs, databases, SaaS applications, files and spreadsheets into Databricks.

• Develop scalable data models and Delta Lake architecture for reporting, analytics and future AI use cases.
• Manage Databricks Workflows, Lakeflow or Delta Live Tables, SQL Warehouses and related processing jobs.
• Administer Databricks workspaces, Unity Catalog, permissions, service principals.
• Support development, test and production environments, including deployment, monitoring, incident resolution and recovery.
• Monitor platform performance, usage and costs, and implement appropriate security and operational controls.
• Implement data quality checks covering completeness, accuracy, timeliness, consistency, duplicates and reconciliation with source systems.
• Ensure dashboards use governed, reliable and well-documented data.
• Establish data ownership, common definitions, lineage, access policies and governance standards with business functions.
• Build monitoring and reporting to identify pipeline failures and data quality issues.
• Integrate governed Databricks data with Power BI and other approved analytical tools.
• Work with Finance, Delivery, Operations, HR and PMO to translate requirements into sustainable data solutions.
• Document data pipelines, controls, dependencies and operating procedures.
• Coordinate external partners where specialist support is required.

• Significant experience as a Data Engineer, including ownership of production data pipelines.
• Strong hands-on experience with Databricks, Delta Lake, Python, SQL and PySpark.
• Experience integrating data from APIs, business applications, databases and files.
• Experience with Unity Catalog, access management, data governance and data quality controls.
• Experience administering Databricks workspaces, compute resources and SQL Warehouses.
• Experience with Azure data services, Power BI, Git and CI/CD practices.
• Understanding of environment management, monitoring, troubleshooting and cost control.
• Ability to work directly with non-technical stakeholders and explain data issues clearly.
• Strong documentation, prioritisation and problem-solving skills.

Certifications (Preferred)

• Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).