Lead Data Engineer - HM Treasury - G7

If you're interested in making a difference to people's lives, the Treasury can offer you an exciting opportunity to influence decision making that affects the whole of the UK. Working at the heart of government, we collaborate across government to promote responsible public spending and drive strong and sustainable economic growth.

Our work ranges from protecting customers through the regulation of the financial sector, helping to reduce carbon emissions and creating a greener economy, to promoting British trade around the world and supporting people across the country on jobs, growth and more.

We are part of the , a cross-government hub bringing people together to tackle key national issues while working closer to the communities we serve.Darlington Economic Campus, As a lead data engineer, you will work on a variety of data engineering projects, maintaining and developing them in response to user feedback. These products are used by a wide variety of HMT staff: improving them will increase productivity across the department. You will also work in a team of data professionals, supporting HMT through a wide portfolio of projects, and hold relationships with customers within HMT and beyond., * You will be acquiring data sets from multiple sources at varying levels of maturity. You will have a demonstrable understanding of how to expose data from systems (e.g. from APIs), link data from multiple systems, and deliver streaming services. You will collaborate with others to identify and plan these services, and to review specifications.

• You will maintain and develop data products used internally and in line with HMT's data strategy. You will leverage your expertise in Python and cloud systems to achieve this. You will ensure the products run reliably and react to any issues rapidly.
• You can produce data models and understand where to use different types of data models. You have experience of a wide range of data engineering tools. You understand industry-recognised data-modelling patterns, how to establish and use agreed standards and tools, design, code, and test programs and scripts for data engineering. You can develop resilient, scalable, and future-proof services for to meet user needs.
• You will actively scope and carry out other projects including web app development and data engineering. You will explore opportunities to work innovatively, recognising, identifying and exploiting opportunities to ensure efficient and effective performance of the organisation, and to support and enable that you will influence the data architecture in place. You can explore new ways of conducting business and organisational processes.
• You will promote best practice in data engineering including acquisition, storage and sharing of data for the benefit of modelling and policy development, working with the Hub to provide overall technical leadership and development opportunities across HMT in data and data science.
• You will contribute to the growth of the Data Hub through sharing work, pair programming, contributing process ideas.

Strong experience using Python for data engineering and web app development - Expertise in SQL - Considerable experience at least one major Cloud provider (Azure is preferred, however Google Cloud or AWS are acceptable) - Experience successfully implementing a product and managing customer relationships, acting on feedback and other good product management practices, Experience making pipelines and data warehousing with Microsoft Fabric - Experience making apps with Plotly Dash, Streamlit and/or Chainlit - Experience using LLM and Embedding APIs, with a strong understanding of how both work, * Proven experience designing and building robust back-end systems

• Demonstrable experience applying technical governance and controls
• Strong ability to translate organisational or user needs into effective technical solutions
• Ability to provide credible technical leadership and advice