Data Engineer | Active SC Required | £550 per day Inside IR35 | Occasional Office Days in London

We are seeking a strong, hands-on Data Engineer to join our engineering team working on the HMPPS Data Hub project. We are building a modular ingestion platform that utilises reusable infrastructure and automated workflows, enabling new data services to be onboarded seamlessly via YAML configuration. In this role, you will focus on designing and building robust, maintainable data pipelines and reusable infrastructure components. You will work across the full ingestion life cycle: from full-load and Change Data Capture (CDC) streaming to schema validation and cloud storage. Beyond technical implementation, you will be a pragmatic problem solver who thinks critically about system retries, failure recovery, end-to-end troubleshooting across distributed services and clear technical communication., * Reusable Ingestion Pipelines: Production-ready ETL/ELT pipelines and AWS Step Functions workflows configured via YAML specifications.

• Terraform Infrastructure Modules: Well-structured, reusable IaC modules deploying platform components reliably across development, test, and production environments.
• Automated Tests & Technical Documentation: Comprehensive pytest test suites, schema validation rules, and clear technical documentation to support ongoing platform operational maintenance.

Working Arrangements

• AWS Cloud Data Engineering: Strong, hands-on commercial experience building data solutions using core AWS services, including Database Migration Service (DMS) for Full Load & CDC, S3, Lambda, Step Functions, DynamoDB and SQS.
• Infrastructure as Code (Terraform): Proven experience building reusable Terraform modules, managing state effectively and deploying infrastructure safely across multiple environments.
• Python Software Development: Strong Python skills focused on Lambda development, AWS SDK (boto3), robust error handling and automated testing using pytest.
• Databases & CDC Concepts: Solid experience working with relational databases (PostgreSQL and Oracle), writing efficient SQL, managing schema changes and applying CDC concepts.
• Data Contracts & Formats: Experience defining and working with data contracts, schema validation, versioning and file formats such as Avro and Parquet.
• CI/CD & Version Control: Proficiency with Git, pull request workflows, GitHub Actions and versioned deployment pipelines.

Technical Experience and Tooling

• Configuration & Orchestration: Experience with YAML/JSON-driven service onboarding and Step Functions workflows.
• Security & Monitoring: Familiarity with AWS IAM, KMS, Secrets Manager and CloudWatch for logging, alerting and secure infrastructure.
• Data Management & Containers: Practical knowledge of Docker/ECR and AWS Glue Data Catalog is highly desirable.