> Markdown version of [/jobs/ext/2301529-consultant-manager-data-engineer-ai-data-defence-security](https://www.wearedevelopers.com/jobs/ext/2301529-consultant-manager-data-engineer-ai-data-defence-security). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Consultant - Manager, Data Engineer, AI & Data, Defence & Security - **Company:** HM Revenue & Customs - **Location:** London, UK - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Airflow, Amazon Web Services, Audit Trail, Microsoft Azure, BigQuery, Cloud Database, Code Reuse, Computer Programming, Continuous Integration, Data as a Services, Data Architecture, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Security, Data Stores, Data Systems, Data Warehousing, Database Design, Database Queries, DevOps, Digital Assets, Distributed Computing Environment, Github, Python (Programming Language), Machine Learning, Metadata, Meta-Data Management, Scala (Programming Language), Software Engineering, Data Streaming, Technical Data Management Systems, Unstructured Data, Workflow Management Systems, Azure Service Bus, Google Cloud, Snowflake, Apache Spark, Gitlab, Containerization, Pyspark, Kubernetes, Apache Flink, Apache Kafka, Apache Nifi, Data Management, Video Streaming, Terraform, Azure Synapse Analytics, Software Version Control, Data Pipelines, Docker, Amazon Redshift, Databricks - **Published:** August 30, 2026 - **Apply:** https://www.collegerecruiter.com/job/2815120062-consultant-manager-data-engineer-ai--data-defence--security ## About the Role All applicants must hold active UK Developed Vetting clearance. We are looking for candidates who combine strong hands-on data engineering capability with the ability to communicate clearly, work collaboratively and explain the value of their work to clients. Essential skills and experience * Hands-on experience designing, building and maintaining data pipelines, ETL/ELT processes or workflow orchestration. * Practical coding experience in Python, Scala, Java or equivalent, with strong SQL skills. * Experience working with diverse data sources and data types, such as batch, streaming, structured, semi-structured or unstructured data. * Understanding of data modelling, database design, data warehousing or lakehouse patterns. * Experience applying software engineering practices such as version control, testing, reusable code, documentation and CI/CD. * Ability to design data solutions that are scalable, reliable, performant and maintainable. * Understanding of data security, access control, data quality, metadata, lineage or broader data governance principles. * Strong communication skills, including the ability to understand user and business needs and translate them into technical data solutions. * Ability to work effectively in multidisciplinary teams and build productive relationships with colleagues and clients. Desirable skills and experience We do not expect candidates to have experience across all the areas below, but we are particularly interested in people who can bring depth in one or more of them: * Distributed data processing using tools such as Spark, PySpark, Flink or equivalent. * Workflow orchestration using tools such as Airflow, Dagster, dbt, NiFi or equivalent. * Cloud-based data engineering services across AWS, Azure, GCP or equivalent platforms. * Data platforms such as Databricks, Snowflake, BigQuery, Redshift, Synapse or similar. * Streaming technologies such as Kafka, Kinesis, Event Hubs or equivalent. * Containerisation, DevOps or platform engineering practices using tools such as Docker, Kubernetes, Terraform or GitHub/GitLab pipelines. * Building data products or reusable data services for analytics, AI or machine learning use cases. * Metadata management, data catalogues, lineage tooling, master data management or data quality frameworks. * Working with sensitive, classified or mission-critical data in secure environments. * Designing architectures that support auditability, resilience, security and operational monitoring. ## Description We are seeking talented data engineers to join our multidisciplinary Engineering, AI and Data team, with a focus on clients in the Defence and Security sector. As a data engineer, you will help clients turn complex, high-volume and sensitive data into trusted, governed and reusable information assets. You will design and build the data pipelines, platforms and products that enable analytics, AI, reporting and operational decision-making. You will work in multidisciplinary teams across a range of client engagements, often in complex and secure environments. Your role may span discovery, architecture, engineering, delivery and ongoing improvement, depending on the needs of the project and your individual skillset. As a practitioner in our team, you will: * Design, build and maintain high-quality data pipelines, data stores and data products. * Ingest, transform, enrich and integrate data from diverse sources, including structured, semi-structured, streaming and unstructured data. * Create trusted, standardised and reusable data assets for downstream use cases such as analytics, AI, reporting and operational services. * Work with engineers, data scientists, analysts, architects and client stakeholders to deliver scalable data solutions. * Apply engineering good practice, including version control, testing, automation, CI/CD and documentation. * Help design data architectures that are secure, reliable, scalable and maintainable. * Support data governance, including metadata, data quality, lineage, access control and appropriate handling of sensitive data. * Monitor, troubleshoot and optimise data systems for performance, reliability and operational use. * Bring innovation and practical problem-solving to complex data engineering challenges. * Contribute to client delivery, stakeholder relationships, practice development and future propositions. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [WeAreDevelopers LIVE - Modern DevOps for IoT Devices and More](https://www.wearedevelopers.com/videos/1805-wearedevelopers-live-modern-devops-for-iot-devices-and-more) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Software Engineer Salary in The UK](https://www.wearedevelopers.com/magazine/231-software-engineer-salary-in-the-uk)