> Markdown version of [/jobs/ext/2838074-data-engineer](https://www.wearedevelopers.com/jobs/ext/2838074-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer - **Company:** Whitehall Resources Limited - **Location:** Cambridge, UK (Remote available) - **Experience:** Experienced - **Contract:** Temporary contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, JIRA, Microsoft Azure, Continuous Integration, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Systems, Data Vault Modeling, Iterative and Incremental Development, IT Management, Python (Programming Language), Power BI, Standard Sql, Search Technologies, SQL Databases, Tableau (Software), Unstructured Data, Workflow Management Systems, Enterprise Data Management, Data Server Interface, Enterprise Software Applications, Microsoft Power Automate, Large Language Models, Multi-Agent Systems, Generative AI, Git, Build Management, Data Lakes, AI Platforms, Pyspark, Kubernetes, Data Pipelines, Servicenow, Databricks - **Published:** September 11, 2026 - **Apply:** https://www.careerjet.co.uk/job/gb5f9c5b3ad06c817f160211279cf609df/eaa ## About the Role * Minimum 4 years' data engineering experience, with strong experience in Databricks. * Strong Python, PySpark and SQL skills. * Experience designing scalable ETL/ELT pipelines and enterprise data models. * Strong understanding of Data Vault, Delta Lake, Lakehouse and Medallion Architecture. * Experience integrating enterprise systems through APIs and other data interfaces. * Experience working with structured and unstructured data. * Practical experience with Generative AI, RAG, embeddings, vector/semantic search and LLM integration. * Experience with AI or workflow orchestration, connecting models, data, APIs and tools. * Experience with automated data quality, testing and monitoring. * Experience with Git and CI/CD and good understanding of data governance and security. Nice to Have Skills and Experience: * Experience with Microsoft Copilot, OpenAI/Azure OpenAI or Microsoft Graph. * Experience with Databricks AI capabilities, including Mosaic AI and Vector Search. * Experience with orchestration frameworks such as LangChain, LangGraph or Semantic Kernel. * Experience developing AI agents or agentic workflows. * Experience with Jira, ServiceNow or Azure DevOps data. * Knowledge of Power BI, Tableau or AI/BI. * Familiarity with Agile delivery and iterative development. * Strong communication and collaboration skills. ## Description We are seeking an experienced Data Engineer to design and deliver scalable, high-quality data solutions in Databricks. You will play a key role in evolving our enterprise data platform, creating trusted, AI-ready data products that power reporting, analytics, automation and AI across Enterprise IT. You will design and build modern data pipelines integrating enterprise platforms such as AI platforms including Microsoft Copilot, OpenAI and other enterprise AI services. The role will also contribute to our growing AI capabilities, engineering solutions for AI usage and adoption data, Retrieval-Augmented Generation (RAG), LLM integration and AI orchestration. Responsibilities: * Design, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python. * Design scalable data models using Data Vault principles. * Build trusted data products using Lakehouse and Medallion Architecture principles. * Integrate data from enterprise systems including ServiceNow, Jira, Azure DevOps and AI platforms such as Copilot and OpenAI. * Build data pipelines for AI usage, adoption, telemetry and performance analytics. * Design and implement RAG pipelines, including ingestion, embeddings, vector/semantic search and retrieval. * Integrate LLMs, APIs and enterprise data to support AI applications and agents. * Build and maintain AI orchestration workflows connecting models, data, tools and business processes. * Implement automated data quality, testing, monitoring and validation. * Optimise solutions for performance, scalability, reliability and cost. * Contribute to data governance, metadata, lineage, security and documentation. * Use Git and CI/CD to support automated and reliable deployment. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)