Data Engineer

This is an exciting role that will allow you to be integral to the evolution of our cloud-based enterprise data platform. As a Data Engineer, you will be working on technical solutions, whilst supporting our existing set of SQL Servers that are predominantly used to feed data into a number of analytics products. This will help the business with data-led decision making to provide essential public services.

Your responsibilities will include:

1. Designing efficient and robust data engineering projects within both on-premises and cloud-based data stacks that add business value and comply with architectural and data security requirements.
2. Building, maintaining and supporting data engineering pipelines, with resolution of incidents.
3. End-to-end delivery of data engineering solutions, including data quality, testing, peer reviewing, documenting and demonstrating of completed solutions to the team.
4. Working closely with teams within data and other technical colleagues across the business.
5. Remaining aware of new data engineering approaches, and be able to suggest how the latest research, techniques and approaches could be implemented to achieve business benefit
6. Undertaking any other duties as required to meet the needs of the business.

1. Substantial experience of data pipeline design and management
2. Enthusiasm to add customer value whilst improving the effectiveness of the data engineering function
3. Proactive problem solving and being able to effectively communicate solutions to both the data team, and to stakeholders
4. Good understanding of database administration activities and principles
5. Data transformation approaches
6. Strong Python and SQL programming capabilities
7. Data security approaches - e.g. permission models (AD, role-based etc.) and techniques to protect data in certain circumstances (e.g. encryption, masking)

Desirable Qualifications, Knowledge and Experience

1. Familiarity with common agile project management principles, particularly in the context of data and analytical project delivery

2. Good understanding of typical ingestion patterns (e.g. ETL, ELT) and their effective implementation with on-premises and cloud-based environments
3. Knowledge of typical data modelling approaches (e.g. Kimball)
4. Experience using the Microsoft data stack (especially Azure Data Lake, Azure Synapse, Azure Data Factory, Databricks)
5. Experience of the full software/reporting lifecycle from planning and design through to deployment and maintenance
6. Knowledge of the Microsoft BI Stack, including Analysis Services, Integration Services and Reporting Services, and Power BI
7. Understanding of DevOps (or other source control technologies) - particularly code repositories and version control, containerisation methodologies, and application deployment practices (CI/CD pipelines)