Senior Data Engineer

The key purpose of this role is to deliver new, and implement changes to existing data solutions, data extraction and transform (ETL), using the full Microsoft SQL stack, for identified and agreed business requirements.

In addition, the Senior Data Engineer will be responsible for translating business requirements into Azure Cloud-based big data solutions for our Data Transformation programme. This will include building complex data pipelines, data warehouses and the data lakes, using Azure services such as Azure Data Factory, Azure Data Bricks and Azure Synapse Analytics.

Your responsibilities will include…

• Leading the design, development, delivery and maintenance of technical on-premises SQL and Azure Cloud data engineering solutions, ensuring alignment with business priorities and best practice
• Implementing robust ETL/ELT pipelines using Azure Data Factory, Synapse Pipelines and Azure Databricks (PySpark/Spark SQL) to support batch and streaming data processing
• Designing and delivering real-time and batch data ingestion solutions using Azure Event Hubs, Kafka, Databricks Structured Streaming and Azure Stream Analytics
• Developing and maintaining modern data platforms, including Delta Lake/Lakehouse architectures on ADLS Gen2, curated datasets, dimensional models and Azure Synapse Dedicated SQL Pools
• Analysing current business practices, processes and procedures, identifying opportunities to leverage Microsoft Azure Data & Analytics Services to improve business outcomes and data processing efficiency
• Maintaining the integrity of data architecture, championing high-quality data standards and the effective use of data repositories, relational models and data warehouse solutions to deliver accurate, consistent and reliable data
• Designing and delivering performance enhancements, application migrations, version upgrades and data platform modernisation initiatives, including migrations from on-premises to public cloud environments
• Championing and mentoring colleagues on Data Engineering best practices, while developing reusable code, libraries, patterns and frameworks for Azure cloud-based data warehousing and data pipelines
• Contributing to the development and continuous improvement of database management services, processes and standards relating to the delivery of data solutions

• Extensive experience with Data Engineering / ETL development experience is essential with a regulated environment is essential
• Significant experience with data solution design experience in an MI / BI / Analytics environment (Kimball, lake house, data lake) is essential
• Significant experience using:
• Azure Data Factory, Azure Synapse Analytics, Azure SQL, Azure Data Lake, Datra bricks, Purview and Azure App Service is essential
• Proficiency in Python/SQL/SparkSQL/PySpark is essential
• Strong experience of Azure delivery experience including analysis / development / configuration/ deployments essential
• Expert level technical skill set in as Transact SQL language and relational databases is essential

We offer a competitive base salary depending on experience from £55,000 to £70,000 and a competitive benefits package including:

• Annual discretionary bonus opportunity: up to 17.5%
• 30 days annual leave
• Choose to contribute 3% or 5% pension, with employer contributions of 6% or 8% respectively
• Enhanced family-focused benefits
• Hybrid-working