Senior Data Engineer

The Senior Data Engineer is the technical owner of the ClarusONE data platform. This is a hands-on engineering role with real ownership: alongside building and delivering data pipelines, you will raise the engineering standard of the platform - improving its reliability, scalability, security, and quality and help shape where it goes next.

The platform has grown quickly, and this role is central to establishing the engineering patterns, frameworks, and best practices that will support it at scale. You will bring a technical lens to infrastructure, deployment practices, and architecture, and act as a trusted technical voice within the team.

You will operate within the ClarusONE data governance framework, ensuring sensitive data is properly protected and that data can be traced end to end through its lifecycle. You will work closely with Analytics, Data Science, Software Engineering, and IT/Infrastructure to maintain an accurate, efficient, and compliant data environment - and will communicate confidently with stakeholders across the business, including Finance., Platform Ownership & Engineering Standards

• Take technical ownership of platform components, ensuring quality and consistency
• Establish, apply, and evolve engineering best practices, design patterns, and coding standards
• Define reusable patterns and frameworks across the data platform (including layered/medallion architecture approaches)
• Identify opportunities to improve performance, cost efficiency, scalability, and reliability
• Contribute to the target architecture and long-term platform direction

Data Engineering & Delivery

• Design and build robust, scalable data pipelines (Snowflake, dbt, Airflow, ADF, Python)
• Apply strong practices in test-driven development, monitoring, logging, and observability
• Own more complex technical solutions and problem-solving
• Support production issues and drive root-cause improvements rather than quick fixes

Data Governance, Traceability & Compliance

• Ensure data lineage and traceability across the data lifecycle, so any figure can be traced back to source
• Ensure sensitive data is appropriately protected within the ClarusONE data governance framework
• Work with IT teams to ensure solutions meet governance and compliance requirements (e.g. SOX)
• Understand and manage the implications of production changes within a controlled release process

Infrastructure & DevOps

• Contribute to and maintain Infrastructure as Code (IaC) for the platform (Azure, AKS) in collaboration with the Infrastructure team
• Work within and help mature CI/CD pipelines, deployment workflows, and governance processes
• Continuously improve how the team builds, tests, and deploys data pipelines

Innovation & Platform Evolution

• Bring forward ideas to improve the platform, tooling, and developer experience
• Evaluate emerging technologies and contribute to adoption decisions
• Support platform evolution, including re-architecture and migration activities

Collaboration & Communication

• Support other Data Engineers by sharing best practices and technical approaches
• Collaborate closely with Analytics & Data Insights, Software Engineering, and IT/Infrastructure
• Present technical work and recommendations to a range of audiences, including company-wide forums
• Build strong working relationships with business stakeholders, including Finance, and translate technical concepts into clear, practical terms

(The above statements describe the general nature and level of work being performed in this job. They are not intended to be an exhaustive list of all duties.)

Degree or equivalent and typically requires 7+ years of relevant experience., Bachelor’s degree level or above, * 6+ years’ experience in data engineering or platform engineering, including at least 2 years at senior or lead level

• 4+ years’ experience building and maintaining CI/CD pipelines, deployment workflows, and associated governance processes
• Advanced SQL and Python, with hands-on experience building data pipelines and transformations at scale
• Hands-on experience with a modern cloud data stack, including Snowflake, dbt, Airflow, and Databricks, on a cloud platform (Azure preferred)
• Demonstrated experience delivering data lineage and traceability within a regulated environment (e.g. SOX, financial services, healthcare, or similar)
• Experience contributing to platform migrations, re-architecture, or modernisation programmes
• Proven ability to present technical concepts to non-technical and senior stakeholders, including Finance, and influence decisions through practical recommendations

Additional Knowledge & Skills

Platform & Infrastructure

• Maintaining Infrastructure as Code for cloud-based data platforms
• Exposure to containerised or distributed systems (AKS advantageous), including deployment, scaling, and operational considerations

Orchestration & Version Control

• Operating Airflow at scale, including DAG reliability, scheduling, and failure handling
• Version control, branching strategies, and release processes

Engineering Best Practice

• Test-driven development and automated testing within CI/CD workflows
• Defining and enforcing standardised engineering patterns across a shared codebase
• Operational reliability, including monitoring, structured logging, alerting, and systematic failure recovery

Desirable

• PySpark
• Experience supporting Airflow in a self-managed environment
• Experience in a business building a data platform from the ground up