> Markdown version of [/jobs/ext/1378573-data-engineer-databricks](https://www.wearedevelopers.com/jobs/ext/1378573-data-engineer-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer (Databricks) - **Company:** M&G - **Location:** Stirling, UK - **Contract:** Temporary contract - **Skills:** Computer Programming, Databases, Data Architecture, Information Engineering, Data Governance, Data Systems, Apache Hive, Python (Programming Language), Software Tools, Data Processing, Apache Spark, Pyspark, Data Management, Data Pipelines, Databricks, Programming Languages - **Published:** July 22, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=be57cbf4e1a65e33 ## About the Role * Strong expertise in data engineering concepts, including data pipelines, storage, and transformation. * Proficiency in data engineering tools, programming languages, and database technologies, in particular in Azure Databricks, Spark, Spark SQL and Python/pyspark. * Understanding of data governance, security, and data quality management principles. * Ability to design scalable and efficient data solutions aligned to business needs. * Strong problem-solving capability with a focus on accuracy and performance. * Ability to communicate technical concepts clearly to both technical and non-technical audiences. * Collaborative approach with the ability to influence decisions and guide others. * Experience working in Financial Services or a similar heavily audited environment with an understanding of operating controls and producing audit evidence Essential Skills: * Data pipeline development and optimisation * Data modelling and architecture * Programming and scripting for data processing * Data governance and quality assurance * Stakeholder engagement and technical communication, including at design forums where relevant * Continuous improvement and innovation * Experience working in an audited environment Desirable Skills: * Azure Databricks * Spark, Spark SQL, Python, PySpark * People management * Experience working in UK Financial Services, ideally Life Insurance ## Description A new solution for the validation and transformation of policyholder data used in actuarial modelling and assumption setting is being built. This fixed term role will lead and oversee the development of new streamlined data pipelines. The role will bring the new solution successfully to a state where ongoing maintenance and development is owned by the BAU team. These data systems need to be robust, reliable, efficient, and meet the requirements for use in actuarial modelling and assumption setting. Operating at a Manager or Expert level, the role provides specialist expertise in data engineering, ensuring scalable, high-quality data pipelines and architectures are in place to support the use of policyholder data for actuarial modelling and assumption setting. The role contributes to shaping engineering standards and best practices, ensuring data is accessible, governed, and aligned with business priorities. Main Responsibilities: * Lead the design and build of scalable data pipelines and data architecture to validate and transform data to meet actuarial modelling needs. * Design pipelines so that they can be continually adapted in a controlled manner once in BAU in order to meet emerging business needs. * Ensure data quality, integrity, and consistency through robust validation, monitoring, and governance practices. * Collaborate with stakeholders to understand data requirements and translate them into efficient and effective technical solutions. * Optimise data systems and processes to improve performance, reliability, and scalability. * Provide technical expertise and guidance on data engineering approaches, tools, and best practices. * Support integration of multiple data sources to create cohesive, accessible, and trusted datasets. * Contribute to the continuous improvement of data platforms, frameworks, and engineering standards. * Lead the work of other data engineers and analysts to ensure consistency and high standards across the team, including people management as required. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Kubernetes and Microservices with Multi-Model Databases](https://www.wearedevelopers.com/videos/382-kubernetes-and-microservices-with-multi-model-databases) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [Software Engineer Salary in The UK](https://www.wearedevelopers.com/magazine/231-software-engineer-salary-in-the-uk) - [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)