Senior Automation Tester (Python) - Agentic AI & LLM Systems (Hybrid)

• Develop and maintain a Python-based automation framework for validating AI agents (chatbots, voicebots, agentic workflows).
• Design and execute automated tests for:
• LLM responses
• Conversational flows
• Retrieval-augmented generation (RAG)
• End-to-end agentic scenarios
• Apply modern AI testing strategies and evaluation methods (e.g., DeepEval, RAGAS - beneficial but not essential).
• Integrate automated tests into CI/CD pipelines (Jenkins, GitHub Actions, Azure Repos).
• Build and manage batch and scheduled executions using Docker, Kubernetes, cron jobs, or similar tooling.
• Manage code repositories using Git, including branching strategies, version control, and collaborative workflows.
• Collaborate closely with engineering, AI/ML, and product teams to ensure high-quality delivery of AI-powered features.

We are seeking a Senior Automation Engineer / SDET with strong Python skills and hands-on experience testing AI systems, including LLM-driven applications and agentic AI workflows. You will design and build a scalable automation framework, implement modern AI testing strategies, and ensure the quality and reliability of next-generation conversational and agent-based AI products.

This role is ideal for someone who is passionate about automation engineering and has a solid understanding of how AI/LLM systems behave in real-world scenarios., * Strong Python development skills, particularly for automation (PyTest preferred).

• Solid background in test automation and framework design.
• Experience testing or validating AI/LLM-based systems (agentic workflows, conversational AI, RAG pipelines, etc.).
• Understanding of AI evaluation concepts - familiarity with tools like DeepEval or RAGAS is a bonus, not a requirement.
• Experience implementing CI/CD pipelines and automated batch executions.
• Strong understanding of source code management, Git workflows, and collaborative engineering practices.
• Ability to design structured, scalable, and maintainable automation solutions.

Preferred Qualities

• Quick learner with a proactive, ownership-driven mindset.
• Strong problem-solving skills and ability to work in fast-moving AI environments.
• Comfortable collaborating with cross-functional teams (AI/ML, engineering, product).
• Passion for AI technologies and modern testing methodologies.