Lead Test Engineer

We are looking for a Test Engineer to join a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB.

You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices., Automated Testing

• Design, implement, and maintain automated test suites across data pipelines, cloud services, and applications
• Automate data and application testing tasks and build test coverage using existing or new infrastructure

Risk Identification & Reporting

• Identify and raise awareness of risks arising from automated test results
• Analyse and report on test activities, results, issues, and risks
• Translate technical findings into clear, prioritised communication

Framework Development

• Identify, evaluate, and implement new test frameworks
• Enhance existing frameworks to improve testing confidence and coverage

Pipeline & Data Testing

• Validate data pipelines using Apache Airflow, AWS Glue, Athena, and EMR
• Ensure data integrity, transformation accuracy, and performance under load
• Analyse data in multiple formats to validate new functionality
• Perform production data analysis to identify root causes of issues

Mentoring & Coaching

• Mentor and advise team members on testing best practices
• Contribute to a quality-first engineering culture

CI/CD & DevOps

• Integrate and manage automated tests within GitLab CI/CD pipelines
• Ensure automated execution and fast feedback loops

Technology Stack

Apache Airflow, Amazon Athena, AWS S3, AWS Glue, AWS EMR, AWS EC2, AWS ECR, AWS DynamoDB, AWS CloudWatch, AWS IAM, Python, Java, SQL, Bash, GitLab CI/CD, Jupyter Notebooks, Apache Spark, Terraform, Docker

• Strong proficiency in Python, Java, SQL, and Scripting (eg, Bash)
• Experience with Apache Airflow for orchestration and log analysis
• Hands-on experience with AWS services (S3, Glue, Athena, EMR, CloudWatch, IAM, DynamoDB, EC2, ECR)
• Experience using GitLab and GitLab CI/CD pipelines
• Ability to analyse data using Jupyter Notebooks and Amazon Athena
• Understanding of Apache Spark and EMR (basic to intermediate)
• Experience with infrastructure as code (Terraform)
• Familiarity with Docker and containerization
• Knowledge of Server Side and client-side encryption
• Understanding of dimensional data models and slowly changing dimensions
• Experience in data creation/generation for testing
• Strong analytical skills for validating data and identifying issues
• Ability to understand how customer requirements translate into functional solutions

Skills & Experience Required

• Proven experience as a Test Engineer or QA Engineer in cloud/data environments
• Strong Python skills for automation, data validation, and test development
• Experience testing data pipelines (data quality, transformation, performance)
• Experience designing or extending automated test frameworks
• Strong problem-solving and root cause analysis skills
• Ability to clearly communicate risks to technical and non-technical stakeholders
• Experience mentoring or coaching junior engineers
• SFIA Level 4 capability - works autonomously and influences others
• Active SC clearance (mandatory)

Scrumconnect Consulting is a UK-based digital transformation consultancy delivering agile, secure technology solutions for public and private sector clients. This is a fully remote role based in India, supporting our UK operations and client base. You will work closely with our UK leadership team and must be comfortable operating within UK regulatory and legal frameworks.