> Markdown version of [/jobs/ext/2786322-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/2786322-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** RedCompass Labs - **Location:** Greater London, UK - **Experience:** Expert - **Salary:** £20,800.0 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Microsoft Azure, Cloud Storage, Encodings, Continuous Integration, Data Architecture, Data Validation, Information Engineering, Data Infrastructure, DevOps, Github, Python (Programming Language), Operational Databases, Parsing, Regression Testing, Search Technologies, Azure Data Factory, Large Language Models, Generative AI, Fastapi, Containerization, Kubernetes, Virtual Agents, Data Pipelines, Docker - **Published:** September 8, 2026 - **Apply:** https://www.collegerecruiter.com/job/2840673762-senior-data-engineer ## About the Role * 5+ years of professional data engineering experience, with a strong track record of designing and operating production data pipelines. * Deep hands-on experience building ingestion and processing pipelines for unstructured and semi-structured content (documents, transcripts, structured data), including parsing, chunking, and metadata enrichment. * Strong proficiency in Python. * Hands-on experience with embedding models, Retrieval-Augmented Generation (RAG) architectures, and vector search (e.g. Azure AI Search, pgvector, or equivalent). * Strong working knowledge of the Azure data stack - Data Factory or Synapse, Functions, ADLS/Blob, CosmosDB, AI Search - and comfort with containerised workloads. * Comfortable making architectural decisions on incomplete information, and willing to revise them as the pipeline meets real data. * A 't-shaped' engineering mindset, and a willingness to stretch into adjacent work - DevOps, retrieval evaluation, prompt tuning, backend APIs - when the work needs it., * Experience with document processing at scale - OCR, layout-aware parsing, or table extraction from PDFs. * Background in fintech, banking, payments, or compliance industries. * Experience building evaluation tooling for retrieval quality (recall@k, faithfulness, regression tests on a curated eval set). * Pipeline & processing core: Python, orchestration frameworks such as Azure Data Factory, Airflow, Dagster, etc. * Retrieval-Augmented Generation (RAG) systems and Azure AI Search for vector and hybrid retrieval. * Storage & data platform: Azure Blob/ADLS, CosmosDB, Azure AI Search, with FastAPI for the retrieval API layer. * Ops & quality: GitHub Actions, Azure DevOps, Docker, with observability and data quality checks across the pipeline. ## Description We're seeking an exceptional Senior Data Engineer to design, build, and own the data pipeline that powers our payments expert agent. Your role: take raw, messy domain content - payments handbooks, regulatory documents, scheme rulebooks, project delivery history - and turn it into a production-grade retrieval layer that an AI agent can reason over reliably. You'll own this end-to-end, from ingestion through to the RAG interface, in a regulated, high-stakes environment., You will own the data pipeline that turns regulated payments domain content into a reliable retrieval layer for our AI agent: * Design, build, and scale the ingestion and processing pipelines that move payments domain content - handbooks, scheme rulebooks, regulatory documents, project history - into structured, retrievable knowledge. * Engineer robust Retrieval-Augmented Generation (RAG) pipelines including chunking, embedding, vector storage, and retrieval, tuned for dense regulatory and technical content. * Stand up cloud-native data infrastructure on Azure - Data Factory, Functions, Blob/ADLS, CosmosDB, and AI Search - with Python as the primary language. * Embed engineering rigour into the pipeline - CI/CD with GitHub Actions and Azure DevOps, containerisation, observability, data quality checks, and re-runnable workflows. * Collaborate closely with the AI Engineer, Data Architect, and payments subject matter experts to translate messy domain content into a retrieval layer the agent can reason over. ## Related Videos - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)