Senior Data and AI Engineer

RES Limited

Glasgow, UK

11 days ago

Apply on www.collegerecruiter.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

10 years minimum

Working hours

Regular working hours

Job source

www.collegerecruiter.com

Tech stack

Artificial Intelligence Airflow Automation of Tests Microsoft Azure Big Data Code Generation Continuous Integration Information Engineering Data Infrastructure Dataspaces Data Structures Data Warehousing

+22 more

Python (Programming Language) Machine Learning Enterprise Messaging Systems Operational Databases Query Optimization Software Tools Software Engineering SQL Databases Management of Software Versions Feature Engineering Microsoft Power Automate Azure Data Factory Large Language Models Microsoft Fabric Data Lakes Scikit Learn Infrastructure Automation Frameworks Information Technology Machine Learning Operations Azure Synapse Analytics Data Pipelines Docker

Job description

Design, build, and operate reliable, secure, and observable data pipelines and curated datasets that power enterprise reporting, analytics, and AI/ML use cases.
Lead AI/ML engineering as a core workstream, designing feature-ready datasets, model pipelines, and AI-ready data products, and applying engineering rigour (testing, versioning, observability) to ML pipelines.
Engineer data products and pipelines that support LLM and generative AI use cases, including retrieval-ready data structures and pipelines feeding AI applications.
Drive engineering automation as a standing discipline, evaluating and adopting AI-assisted code generation and testing tools to reduce manual engineering effort, and building internal tooling and patterns the wider team can use to build faster.
Own engineering quality, performance, and cost optimisation across the platform, implementing data quality controls, testing frameworks, monitoring, and observability.
Build and maintain production-grade data infrastructure on Azure / Microsoft Fabric, including data lakes, lakehouses, and modern data warehouse patterns.
Define and implement CI/CD pipelines for data and ML engineering workflows, applying infrastructure-as-code and automated quality gates as standard practice.
Apply containerisation and orchestration tooling (e.g. Docker, Airflow, or equivalent) to production data and ML workflows.
Partner across architecture, governance, data modelling, and reporting to deliver coherent, end-to-end data and AI products.
Mentor and support engineers, setting the standard for quality, craft, and engineering rigour, including how the team uses AI-assisted automation.
Contribute engineering expertise to RES’s Synapse-to-Fabric migration programme, working alongside the platform architect to convert pipelines and warehouse objects at scale using AI-assisted tooling.

Requirements

Previous experience as a data engineer and senior engineer. Typically 10+ years.
Azure Fabric data platform - expertise across Azure Data Factory, Synapse, Microsoft Fabric, Purview, Unity Catalogue, and Data Lake / Lakehouse architectures.
Python - advanced proficiency including open-source data and ML libraries, frameworks, and production pipeline development.
SQL - expert-level for data modelling, transformation, and complex query optimisation.
AI/ML engineering - building data infrastructure for machine learning and AI use cases, feature engineering, model pipeline support, and production ML pipeline engineering.
AI-assisted engineering automation - experience using AI coding and conversion tools (e.g. Copilot, Claude, or equivalent) to accelerate engineering work at scale, with a clear approach to validating their output.
MLOps - CI/CD for data and ML pipelines, infrastructure as code, containerisation, and orchestration tools such as Airflow or equivalent.
Data quality & observability - hands-on experience with testing frameworks, monitoring, and quality controls in production environments.
LLMs and generative AI - practical understanding of how to engineer data products and pipelines that support LLM and GenAI use cases.
Technical leadership - track record of engineering and architectural decision-making across data and AI/ML disciplines, setting standards, and delivering automated engineering work while contributing to strategy and roadmap thinking., * Degree in computer science, data engineering, software engineering, or a related field - or equivalent hands-on experience.
Significant experience (typically 10+ years) delivering enterprise-grade data engineering solutions in production environments, with meaningful experience in ML/AI engineering.
Proven track record as a Senior Data Engineer or Senior Data & AI/ML Engineer, including building large-scale data and ML systems.
Deep expertise in the Microsoft Azure data ecosystem - ADF, Synapse, Fabric, Purview, Unity Catalogue.
Advanced Python skills including open-source data and ML libraries, frameworks, and messaging systems.
Strong experience building and maintaining production data infrastructure for AI and ML consumption, including model pipelines and feature engineering.
Experience with MLOps practices: CI/CD for data and ML pipelines, automated testing, and infrastructure as code.
Experience with modern data stack tooling - dbt, Airflow, Prefect, or equivalent orchestration and transformation frameworks.
Experience with automation tooling such as Power Automate, Power Platform, or equivalent, and practical use of AI-assisted engineering tools in production settings.
Relevant certifications in Microsoft Azure, data engineering, or AI/ML.
Exposure to working alongside data scientists and AI engineers in a shared platform model., * Experience with a platform migration at scale - such as Synapse to Fabric or an equivalent large data platform transition - using automation or AI tooling to accelerate conversion work.