> Markdown version of [/jobs/ext/5432-automation-qa-engineer](https://www.wearedevelopers.com/jobs/ext/5432-automation-qa-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Automation QA Engineer

- **Company:** Ciklum
- **Location:** Spain (Remote available)
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Microsoft Azure, Databases, Continuous Integration, Information Engineering, Data Files, Python (Programming Language), PostgreSQL, Machine Learning, Performance Tuning, Regression Testing, Systems Integration, Scripting, Large Language Models, Deployment Automation, Data Analytics
- **Published:** May 17, 2026
- **Apply:** https://es.indeed.com/viewjob?jk=ceb236c1c1ac3842

## About the Role

Do you have experience in Python?, Do you have a Master's degree?, * Background: Mid-to-Senior level experience in Data Science, Machine Learning Evaluation, AI Quality Assurance, or Data Engineering

 * Evaluation Frameworks: Deep, hands-on experience with LLM evaluation frameworks (e.g., RAGAS, DeepEval, TruLens) and establishing human-anchored or synthetic benchmarks
 * Technical Stack: Strong proficiency in Python. Solid experience with CI/CD tools (especially Azure DevOps) and integrating complex test suites into automated deployment pipelines
 * Data & Observability: Experience working with databases (PostgreSQL) and integrating custom telemetry or observability data (e.g., Azure App Insights) into evaluation reports
 * Analytical Mindset: Strong attention to detail with the ability to perform rigorous error analysis, build structured taxonomies for failures, and identify embedding drift

Personal skills:

 * Highly collaborative and data-driven; comfortable working directly with client SMEs to validate queries and presenting evaluation scorecards to guide engineering decisions

## Description

As a Automation QA Engineer, become a part of a cross-functional development team engineering experiences of tomorrow., * Own the evaluation lifecycle, offline acceptance testing, and KPI measurement for the AS client's RAG pipeline

 * Lead the co-creation and management of the project's "golden dataset" to consistently benchmark AI performance
 * Implement and manage the RAGAS evaluation harness and automated CI/CD regression testing
 * Track, classify, and build root-cause taxonomies for LLM hallucinations, with a specialized focus on code-generation correctness
 * Golden Dataset & Baselines: Collaborate with client domain experts and technical leads to build a robust synthetic test set (~90+ queries across multiple categories) and establish baseline metrics for Faithfulness, Context Precision, and Answer Relevance
 * Evaluation Harness: Build and automate evaluation pipelines using RAGAS and custom Python scripts, enabling A/B comparisons between the baseline, MVP, and full implementation
 * Regression & CI/CD Guardrails: Implement automated CI/CD regression checks within Azure DevOps, ensuring that a >5% drop in core metrics automatically blocks pipeline deployments
 * Hallucination Tracking: Develop a root-cause taxonomy for hallucinations and track code-generation queries separately to ensure the AI generates functionally correct and compilable output
 * Performance Benchmarking: Measure and monitor pipeline latency, rigorously validating P95 latency targets (sub-4.5s) under representative concurrent load, * Strong community: Work alongside top professionals in a friendly, open-door environment
 * Growth focus: Take on large-scale projects with a global impact and expand your expertise
 * Tailored learning: Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications
 * Endless opportunities: Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies
 * Flexibility: Enjoy flexibility - full remote working possibilities
 * Care: We've got you covered with company-paid medical insurance, mental health support, and financial & legal consultations


## Related Videos

- [Kubernetes and Microservices with Multi-Model Databases](https://www.wearedevelopers.com/videos/382-kubernetes-and-microservices-with-multi-model-databases)
- [JavaScript? No. Java Scripts! - Scripting with Java](https://www.wearedevelopers.com/videos/2094-javascript-no-java-scripts-scripting-with-java)
- [Introduction to TXT](https://www.wearedevelopers.com/videos/30-introduction-to-txt)
- [Hiring AI Native Talents](https://www.wearedevelopers.com/videos/100268-hiring-ai-native-talents)
- [Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases](https://www.wearedevelopers.com/videos/1146-fault-tolerance-and-consistency-at-scale-harnessing-the-power-of-distributed-sql-databases)
- [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)
- [The 12 Best Jobs for Software Engineers](https://www.wearedevelopers.com/magazine/401-the-12-best-jobs-for-software-engineers)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)