Test Engineer
Role details
Job location
Tech stack
Job description
About Scrumconnect
Scrumconnect Consulting is a UK-based digital transformation consultancy delivering agile, secure technology solutions for public and private sector clients. This is a fully remote role based in India, supporting our UK operations and client base. You will work closely with our UK leadership team and must be comfortable operating within UK regulatory and legal frameworks.
About the role
Test Engineer on a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices.
This is a hands-on, technically deep role. You will write code, build and extend test frameworks, and take accountability for test quality across releases - while actively mentoring more junior team members in best practices.
Key responsibilities
Automated testing
Design, implement, and maintain automated test suites across data pipelines, cloud services, and application layers. Ensure reproducibility and reliability across all releases.
Risk identification
Identify and raise awareness of risks arising from automated test results. Translate technical test findings into clear, prioritised risk communication for the team and stakeholders.
Framework development
Research, evaluate, and implement new test frameworks suited to the AWS data stack. Continuously improve existing frameworks to increase coverage and confidence in test outcomes.
Pipeline & data testing
Validate data pipelines built on Apache Airflow, AWS Glue, Athena, and EMR - including data integrity, transformation accuracy, and performance under load.
Mentoring & coaching
Mentor and advise team members in testing practices. Share knowledge actively, contribute to a quality-first culture, and support less experienced colleagues in developing test skills.
CI/CD integration
Integrate and manage tests within GitLab CI/CD pipelines. Ensure tests run automatically on commit, providing fast feedback loops and blocking regressions before deployment.
Technology stack
Apache AirflowAmazon AthenaAWS S3AWS GlueAWS EMRAWS EC2AWS ECRAWS DynamoDBAWS CloudWatchAWS IAM PythonJavaSQLBash GitLab CI/CD Jupyter Notebooks
Skills and experience required
- Demonstrable experience as a Test Engineer or QA Engineer working on data pipelines or cloud-native environments
- Strong Python proficiency - used for test Scripting, data validation, and automation in a production context
- Working knowledge of Java and SQL for test development and data querying
- Scripting experience in Bash or equivalent for pipeline and environment automation
- Hands-on experience with the AWS ecosystem - particularly S3, Glue, Athena, EMR, CloudWatch, IAM, and DynamoDB
- Experience configuring and running tests within GitLab CI/CD pipelines
- Proven ability to design and implement automated test frameworks from scratch or extend existing ones
- Experience testing data pipelines - including data quality, integrity, transformation logic, and performance
- Ability to identify, articulate, and communicate testing risks clearly to both technical and non-technical audiences
- SFIA Level 4 capability - works autonomously, takes ownership of test quality, and influences others
- Experience mentoring or coaching junior engineers in testing practices
- Active SC clearance - mandatory at point of application, no exceptions
Diversity and Inclusion
At Scrumconnect Consulting, we believe that diversity drives innovation. We are committed to creating an inclusive environment where every individual is respected, valued, and supported. We welcome applications from candidates of all backgrounds and experiences, and we actively encourage applications from women, people with disabilities, under-represented communities, and those seeking flexible working arrangements.
About the role
Test Engineer on a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB. You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices.
This is a hands-on, technically deep role. You will write code, build and extend test frameworks, and take accountability for test quality across releases - while actively mentoring more junior team members in best practices.
Key responsibilities
Automated testing
Design, implement, and maintain automated test suites across data pipelines, cloud services, and application layers. Ensure reproducibility and reliability across all releases.
Risk identification
Identify and raise awareness of risks arising from automated test results. Translate technical test findings into clear, prioritised risk communication for the team and stakeholders.
Framework development
Research, evaluate, and implement new test frameworks suited to the AWS data stack. Continuously improve existing frameworks to increase coverage and confidence in test outcomes.
Pipeline & data testing
Validate data pipelines built on Apache Airflow, AWS Glue, Athena, and EMR - including data integrity, transformation accuracy, and performance under load.
Mentoring & coaching
Mentor and advise team members in testing practices. Share knowledge actively, contribute to a quality-first culture, and support less experienced colleagues in developing test skills.
CI/CD integration
Integrate and manage tests within GitLab CI/CD pipelines. Ensure tests run automatically on commit, providing fast feedback loops and blocking regressions before deployment.
Technology stack
Apache AirflowAmazon AthenaAWS S3AWS GlueAWS EMRAWS EC2AWS ECRAWS DynamoDBAWS CloudWatchAWS IAM PythonJavaSQLBash GitLab CI/CD Jupyter Notebooks
Skills and experience required
- Demonstrable experience as a Test Engineer or QA Engineer working on data pipelines or cloud-native environments
- Strong Python proficiency - used for test Scripting, data validation, and automation in a production context
- Working knowledge of Java and SQL for test development and data querying
- Scripting experience in Bash or equivalent for pipeline and environment automation
- Hands-on experience with the AWS ecosystem - particularly S3, Glue, Athena, EMR, CloudWatch, IAM, and DynamoDB
- Experience configuring and running tests within GitLab CI/CD pipelines
- Proven ability to design and implement automated test frameworks from scratch or extend existing ones
- Experience testing data pipelines - including data quality, integrity, transformation logic, and performance
- Ability to identify, articulate, and communicate testing risks clearly to both technical and non-technical audiences
- SFIA Level 4 capability - works autonomously, takes ownership of test quality, and influences others
- Experience mentoring or coaching junior engineers in testing practices
- Active SC clearance - mandatory at point of application, no exceptions
Requirements
- Demonstrable experience as a Test Engineer or QA Engineer working on data pipelines or cloud-native environments
- Strong Python proficiency - used for test Scripting, data validation, and automation in a production context
- Working knowledge of Java and SQL for test development and data querying
- Scripting experience in Bash or equivalent for pipeline and environment automation
- Hands-on experience with the AWS ecosystem - particularly S3, Glue, Athena, EMR, CloudWatch, IAM, and DynamoDB
- Experience configuring and running tests within GitLab CI/CD pipelines
- Proven ability to design and implement automated test frameworks from scratch or extend existing ones
- Experience testing data pipelines - including data quality, integrity, transformation logic, and performance
- Ability to identify, articulate, and communicate testing risks clearly to both technical and non-technical audiences
- SFIA Level 4 capability - works autonomously, takes ownership of test quality, and influences others
- Experience mentoring or coaching junior engineers in testing practices
- Active SC clearance - mandatory at point of application, no exceptions
