> Markdown version of [/jobs/ext/2820715-senior-automation-tester-python-agentic-ai-llm-systems-hybrid](https://www.wearedevelopers.com/jobs/ext/2820715-senior-automation-tester-python-agentic-ai-llm-systems-hybrid). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Automation Tester (Python) - Agentic AI & LLM Systems (Hybrid)

- **Company:** Resillion
- **Location:** Glasgow, UK
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Automation of Tests, Continuous Delivery, Cron, Github, Python (Programming Language), Strategies of Testing, Chatbots, Large Language Models, Git, Build Management, Pytest, Git Flow, Kubernetes, Microsoft Tfs, Virtual Agents, Software Version Control, Docker, SDET, Jenkins
- **Published:** September 10, 2026
- **Apply:** https://www.collegerecruiter.com/job/2840613316-senior-automation-tester-python--agentic-ai--llm-systems-hybrid

## About the Role

We are seeking a Senior Automation Engineer / SDET with strong Python skills and hands-on experience testing AI systems, including LLM-driven applications and agentic AI workflows. You will design and build a scalable automation framework, implement modern AI testing strategies, and ensure the quality and reliability of next-generation conversational and agent-based AI products.

This role is ideal for someone who is passionate about automation engineering and has a solid understanding of how AI/LLM systems behave in real-world scenarios., * Strong Python development skills, particularly for automation (PyTest preferred).

 * Solid background in test automation and framework design.
 * Experience testing or validating AI/LLM-based systems (agentic workflows, conversational AI, RAG pipelines, etc.).
 * Understanding of AI evaluation concepts - familiarity with tools like DeepEval or RAGAS is a bonus, not a requirement.
 * Experience implementing CI/CD pipelines and automated batch executions.
 * Strong understanding of source code management, Git workflows, and collaborative engineering practices.
 * Ability to design structured, scalable, and maintainable automation solutions.

Preferred Qualities

 * Quick learner with a proactive, ownership-driven mindset.
 * Strong problem-solving skills and ability to work in fast-moving AI environments.
 * Comfortable collaborating with cross-functional teams (AI/ML, engineering, product).
 * Passion for AI technologies and modern testing methodologies.

## Description

* Develop and maintain a Python-based automation framework for validating AI agents (chatbots, voicebots, agentic workflows).
 * Design and execute automated tests for:
 + LLM responses
 + Conversational flows
 + Retrieval-augmented generation (RAG)
 + End-to-end agentic scenarios
 * Apply modern AI testing strategies and evaluation methods (e.g., DeepEval, RAGAS - beneficial but not essential).
 * Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, Azure Repos).
 * Build and manage batch and scheduled executions using Docker, Kubernetes, cron jobs, or similar tooling.
 * Manage code repositories using Git, including branching strategies, version control, and collaborative workflows.
 * Collaborate closely with engineering, AI/ML, and product teams to ensure high-quality delivery of AI-powered features.


## Related Videos

- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [From clicks to cribs - How to find your dream home with web scraping](https://www.wearedevelopers.com/videos/767-from-clicks-to-cribs-how-to-find-your-dream-home-with-web-scraping)
- [Testing AI Agents: Automated Evaluation for Chatbots & RAG Systems](https://www.wearedevelopers.com/videos/100300-testing-ai-agents-automated-evaluation-for-chatbots-rag-systems)
- [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews)
- [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing)
- [Is your backend a hodgepodge of queues, event stores and cron jobs? Durable Execution to the Rescue.](https://www.wearedevelopers.com/videos/744-is-your-backend-a-hodgepodge-of-queues-event-stores-and-cron-jobs-durable-execution-to-the-rescue)

## Related Articles

- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)
- [ Dev Digest 213: Petrol Prices, Agentic Workflows, AI Skills and CODE100!](https://www.wearedevelopers.com/magazine/718-dev-digest-213-petrol-prices-agentic-workflows-ai-skills-and-code100)