Lead/Principal Data Engineer

• Develop, test, and maintain robust data pipelines that meet functional and non-functional business requirements.
• Design, build, and optimise data-processing solutions for batch and real-time analytics.
• Contribute to logical and physical database design across a range of data platforms.
• Expand and enhance platform capabilities to address new and evolving data challenges.
• Analyse complex datasets and identify opportunities to improve data processing, quality, and performance.
• Ensure data models align with established data architecture standards, maintaining consistency across entities, relationships, and definitions.
• Perform root cause analysis on data-related issues and implement effective solutions.
• Collaborate with engineers, architects, and stakeholders to deliver scalable, reliable data solutions.
• Support continuous improvement and promote best practices in data engineering and platform development.
• Produce clear technical documentation covering data pipelines, processes, and system enhancements.

Technologies:

• AWS
• Lambda
• Ansible
• CI/CD
• Cloud
• DevOps
• EC2
• Support
• Jenkins
• Linux
• NoSQL
• Python
• Security
• TDD
• Terraform
• Windows, We are seeking a Data Engineer to join our growing, collaborative Agile engineering team, delivering secure, large-scale data solutions for government and public sector customers. You will work with modern cloud technologies and open-source tools on complex platforms that support critical decision-making and operational outcomes. This role offers exposure to innovative cloud and data technologies and the opportunity to contribute to meaningful, secure, mission-critical government programmes. We value respect, equality, and inclusion, welcome applicants from all backgrounds, and are committed to supporting reasonable adjustments during recruitment.

• Current and active SC (Security Check) Clearance at the time of application; candidates with expired clearance or who need to obtain clearance cannot be considered, and sponsorship or clearance applications are not supported.
• Experience developing and maintaining data pipelines using modern data-processing technologies and frameworks.
• Knowledge of Agile delivery methodologies and experience in fast-paced development environments.
• Strong understanding of data modelling, data structures, and their advantages and limitations across different use cases.
• Experience with public cloud platforms, particularly AWS, including EC2, RDS, Lambda, Athena, and Glue.
• Experience configuring and optimising relational and NoSQL databases, including query tuning and performance optimisation.
• Proficiency in programming and scripting languages such as Python.
• Experience applying Test-Driven Development (TDD) methodologies and associated testing frameworks.
• Strong analytical and problem-solving skills, excellent communication skills, and the ability to work effectively in multidisciplinary teams.
• Bachelors degree or equivalent commercial experience in a relevant discipline.
• Desirable: experience with CI/CD tools such as Jenkins or Drone; configuration management tools such as Ansible and Packer; Terraform for Infrastructure-as-Code deployments; Linux and Windows environments; cloud-native architectures and DevOps practices; or government, defence, or other highly regulated environments.
• Attention to detail, commitment to high-quality solutions, a collaborative and proactive approach to learning, and the ability to translate business requirements into scalable data solutions.