Principal AI Data Engineer

Deloitte

London, UK

11 days ago

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Compensation

£110,267.0

Working hours

Regular working hours

Job source

www.adzuna.co.uk

Tech stack

Artificial Intelligence Data Analysis Microsoft Azure Code Reuse Computer Programming Continuous Integration Data Architecture Extract Transform Load (ETL) Software Design Patterns DevOps Github Python (Programming Language)

+25 more

Network Planning and Design Object-Oriented Software Development Open Source Technology Performance Tuning Scrum Methodology Rapid Prototyping Process SQL Databases Data Processing Data Ingestion Pytorch Large Language Models Prompt Engineering Generative AI Git Data Lakes AI Platforms Qlikview Enterprise Integration Code Inspection Spark Streaming Machine Learning Operations Video Streaming GPT Docker Databricks

Job description

Develop and evaluate AI/GenAI/AgenticAI prototypes using tools like Copilot Studio, AI Foundry and Copilot Analyst Agent, Mosiac AI, Genie, AgentBricks, MLflow with a focus on quick wins and enterprise integration.
Build and tune Retrieval-Augmented Generation (RAG) systems, including embedding model selection, prompt engineering, and traceable evaluation.
Design and deploy basic AI agents using frameworks such as LangChain, AutoGen, and smolagents
Communicate complex AI concepts clearly to business stakeholders and cross-functional teams.
Collaborate on E platform enhancements and work within its current limitations.
Deploy models and applications using Azure OpenAI, Azure AI Foundry, Databricks Mosaic Gateway, and Docker.
Follow DevOps best practices including CI/CD pipelines, testing, linting, and GitHub workflows.
Write modular, reusable code using OOP design patterns in Python (Pydantic, PyTorch, etc.).
Operate in agile teams and contribute to sprint planning, reviews, and retrospectives.
Deliver hands on GenAI/AgenticAI systems used directly by commercial teams within Trading & Supply, taking solutions from prototype to production
Apply engineering skills (emphasis on Databricks) and research skills across experimentation, rapid prototyping, and iterative delivery. Someone who puts emphasis on reproducibility and open source, manages large-scale text and structured datasets on Databricks.
Build AI capability, manage stakeholders and communicate effectively to ensure alignment between business needs and AI solutions, and a quick understanding of commercial operations that happen in T&S
Design and run evaluation and testing frameworks for GenAI systems, including benchmarking, reproducibility checks, and structured model assessments
Build solutions using Databricks infrastructure, Genie, MLflow (deployment and tracing and evaluations), LangChain, and LangGraph, and integrate them into scalable AI workflows and architectures
Contribute to system planning, architectural design, and structured testing to ensure long term reliability, performance, and maintainability
Preferably also someone who can set the building blocks and lead building out the backlog

Requirements

Bachelor or Master or equivalent in Statistics, Mathematics, Econometrics or similar discipline with at least 8-12 years’ experience on data science/AI projects.
Deep understanding of LLM families (GPT, Llama, Claude, Mistral) and their reasoning capabilities.
Strong experience with Databricks- DLT, Delta Lake concepts, UC governance.
Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
Programming skills in Python, SQL, or Scala.
Proficiency in data modelling, ETL/ELT processes, and data architecture.
Strong analytical background with problem-solving skills.
Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing.
Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication.
Experience working in Azure cloud environments.
Experience with GenAI evaluation frameworks and benchmarking methodologies.
Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
Strong Git practices and collaborative coding standards.
A passion for and expertise in practicing data science to solve real-world problems.
Excellent oral and written communication skills.
Strong interpersonal skills and enthusiasm for teamwork, as well as the ability to work independently.
Familiarity with the enterprise AI platforms and governance models is a plus.
Strong decision-making abilities, using data-driven insights to make informed choices that align with organizational goals.
Skills in managing conflicts and facilitating effective resolutions to maintain a positive and productive team dynamic.
Ability to engage with and manage expectations of various stakeholders, including executives, project managers, and other teams.
Proficiency in identifying potential risks in data projects and implementing strategies to mitigate them.
Strong commitment and ownership of project delivery.