Senior Data Engineer

Our team is looking for a Senior Data Engineer to join our Data Platform team and help deliver our product strategy for managing and analysing social research data. You will work primarily with Azure and Databricks, supported by technologies like Power BI, to design, build, and maintain scalable, reliable, and high-quality data pipelines.

As a key member of the team, you'll collaborate closely with product managers, lead engineers, and cross-functional teams to turn business requirements into technical solutions that drive real-world impact., * Design, develop and maintain data pipelines using Azure Data Factory (ADF), Databricks, Synapse, Python, R, Spark, Pandas and SQL.

• Manage and optimize Azure resources including Azure Portal, Blob Storage, and Data Lake.
• Implement and manage data workflows, ensuring seamless data orchestration and integration.
• Develop and maintain ETL processes, including delta loading and change data capture (CDC) mechanisms.
• Implement pipelines with Azure DevOps, Terraform and/or CI/CD.
• Ensure data security, quality, and governance across all, including access controls and retention policies.
• Collaborate with stakeholders to translate business requirements into technical specifications.
• Utilize Power BI (PBI) for data visualization and reporting.

Do you have experience in Spark?, * Proven experience implementing data platforms and products.

• Strong experience with Cloud technologies, particularly Azure.
• Proficiency in Python, R, Spark, Pandas, SQL.
• Experience working with Delta Live Tables.
• Familiarity with Databricks medallion architecture.
• Experience with Agile and DevOps methodologies.
• Knowledge of Unity Catalog.
• Experience with CI/CD pipelines and Terraform.

Joining Verian means becoming part of a dynamic and collaborative community that thrives on innovation and intellectual curiosity.

• Play a highly impactful role in building a data platform from the ground up, right at the heart of our digital transformation.
• Contribute to the creation of innovative data products leveraging AI and Machine Learning.
• Work on cutting-edge projects that influence public policy and drive meaningful social change.
• Enjoy flexible working arrangements that support work-life balance.
• Collaborate with a passionate, and supportive team in an environment where wellbeing truly matters.

Verian works with clients around the world, providing evidence, insights and advisory services to inspire the next generation of public policy and programmes. We operate at the intersection of data, technology, and human understanding. With permanent offices in 17 countries and a team of 800 specialist consultants and researchers, we advise government departments, regulators, charities, and public sector bodies to make better decisions and improve people's lives.