Principal AI Data Engineer

This role is an initial contract until the end of the year and is hybrid, requiring 3 days per week onsite in London.

You’ll be responsible for:

• Developing AI, GenAI and Agentic AI prototypes using Azure AI Foundry, Copilot Studio, Databricks Mosaic AI and MLflow.
• Building, optimising and evaluating Retrieval-Augmented Generation (RAG) solutions using prompt engineering and embedding models.
• Designing and deploying AI agents using frameworks including LangChain, LangGraph and AutoGen.
• Delivering production-ready GenAI applications for commercial teams within Trading & Supply.
• Building scalable AI workflows on Databricks using Mosaic AI, MLflow, Genie and AgentBricks.
• Applying DevOps best practices through CI/CD pipelines, GitHub workflows and automated testing.
• Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability.

• Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow.
• Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral.
• Experience building and deploying GenAI, RAG and Agentic AI solutions in enterprise environments.
• Strong programming skills in Python, SQL and/or Scala.
• Experience with Azure cloud technologies including Azure OpenAI and Azure AI Foundry.
• Knowledge of Spark Structured Streaming, Autoloader and modern data engineering practices.
• Experience with ETL/ELT, data modelling and scalable data architectures.
• Familiarity with Azure Data Factory, Qlik Replicate and data ingestion pipelines.