Data Engineer (Databricks)

A new solution for the validation and transformation of policyholder data used in actuarial modelling and assumption setting is being built.

This fixed term role will lead and oversee the development of new streamlined data pipelines. The role will bring the new solution successfully to a state where ongoing maintenance and development is owned by the BAU team. These data systems need to be robust, reliable, efficient, and meet the requirements for use in actuarial modelling and assumption setting.

Operating at a Manager or Expert level, the role provides specialist expertise in data engineering, ensuring scalable, high-quality data pipelines and architectures are in place to support the use of policyholder data for actuarial modelling and assumption setting. The role contributes to shaping engineering standards and best practices, ensuring data is accessible, governed, and aligned with business priorities.

Main Responsibilities:

• Lead the design and build of scalable data pipelines and data architecture to validate and transform data to meet actuarial modelling needs.
• Design pipelines so that they can be continually adapted in a controlled manner once in BAU in order to meet emerging business needs.
• Ensure data quality, integrity, and consistency through robust validation, monitoring, and governance practices.
• Collaborate with stakeholders to understand data requirements and translate them into efficient and effective technical solutions.
• Optimise data systems and processes to improve performance, reliability, and scalability.
• Provide technical expertise and guidance on data engineering approaches, tools, and best practices.
• Support integration of multiple data sources to create cohesive, accessible, and trusted datasets.
• Contribute to the continuous improvement of data platforms, frameworks, and engineering standards.
• Lead the work of other data engineers and analysts to ensure consistency and high standards across the team, including people management as required.

• Strong expertise in data engineering concepts, including data pipelines, storage, and transformation.
• Proficiency in data engineering tools, programming languages, and database technologies, in particular in Azure Databricks, Spark, Spark SQL and Python/pyspark.
• Understanding of data governance, security, and data quality management principles.
• Ability to design scalable and efficient data solutions aligned to business needs.
• Strong problem-solving capability with a focus on accuracy and performance.
• Ability to communicate technical concepts clearly to both technical and non-technical audiences.
• Collaborative approach with the ability to influence decisions and guide others.
• Experience working in Financial Services or a similar heavily audited environment with an understanding of operating controls and producing audit evidence

Essential Skills:

• Data pipeline development and optimisation
• Data modelling and architecture
• Programming and scripting for data processing
• Data governance and quality assurance
• Stakeholder engagement and technical communication, including at design forums where relevant
• Continuous improvement and innovation
• Experience working in an audited environment

Desirable Skills:

• Azure Databricks
• Spark, Spark SQL, Python, PySpark
• People management
• Experience working in UK Financial Services, ideally Life Insurance

Pulled from the full job description

• Annual leave
• Company pension
• Private medical insurance, * A valuable pension scheme of up to 18% (13% made up of employer contributions and 5% employee contributions).
• Access to our Share Save and Share Incentive Plan, alongside financial wellbeing and support services to help give you real confidence to put your money to work.
• Enjoy 38 days annual leave (including bank holidays), with the opportunity to purchase up to five extra days. Our Time Off When You Need It policy gives you the flexibility - to balance work and personal commitments.
• Our market-leading Inspiring Families policy includes comprehensive support and paid parental leave covering maternity, adoption, surrogacy, and paternity leave - because supporting families is an important part of our inclusive culture.
• Health & Protection cover includes Private Healthcare, Critical Illness cover and Life Assurance for you, with additional family options - for peace of mind.

At M&G, our purpose is to give everyone real confidence to put their money to work. With a heritage dating back more than 175 years, we have a long history of innovation in savings and investments, combining asset management and insurance expertise to offer a wide range of solutions. Our two distinct operating segments, Asset Management and Life, work together to provide access to balanced, long-term investment and savings solutions. Through telling it like it is, owning it now and moving it forward together with care and integrity; we are creating an exceptional place to work for exceptional talent. We will consider flexible working arrangements for any of our roles and offer workplace adjustments to ensure you have the support you need to succeed in your role.