> Markdown version of [/jobs/ext/1197448-ai-test-automation-engineer](https://www.wearedevelopers.com/jobs/ext/1197448-ai-test-automation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Test / Automation Engineer - **Company:** Cadence, Inc. - **Location:** United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Amazon Web Services, Automation of Tests, Microsoft Azure, Bash Shell, Cloud Computing, Code Coverage, Computer Programming, Continuous Integration, Github, Python (Programming Language), Regression Testing, Prometheus, Selenium, Software Deployment, Software Engineering, Software Testing Automation Framework, Strategies of Testing, TypeScript, Google Cloud, Large Language Models, Grafana, Multi-Agent Systems, Cypress (Programming Language), Reliability of Systems, Pytest, Gitlab-ci, Kubernetes, Information Technology, Playwright, Machine Learning Operations, SDET, Jenkins - **Published:** July 7, 2026 - **Apply:** https://careers.asaltech.com/o/ai-test-automation-engineer ## About the Role * Strong programming skills in Python; familiarity with Bash, TypeScript, or Go * Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress * Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI) * Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes) AI / ML & Agentic Systems * Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock) * Familiarity with: + RAG architectures and vector databases (Pinecone, Weaviate) + Agent frameworks (LangChain, LlamaIndex, AutoGen) AI Testing Techniques * Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds) * Knowledge of evaluation methods: + LLM-as-a-judge + BLEU, ROUGE, semantic similarity scoring * Experience with prompt and agent regression testing * Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection Tooling (Preferred) * AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases * Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard * Monitoring: Prometheus, Grafana, OpenTelemetry Soft Skills * Strong analytical and problem-solving skills * Excellent communication and cross-functional collaboration * Data-driven mindset with focus on quality KPIs * Detail-oriented with a strong bias toward automation and scalability, * 4-7+ years in QA, SDET, or test automation engineering. * Bachelor's or Master's degree in Computer Science, Software Engineering, or related field * Proven experience building and scaling automation frameworks * Hands-on experience with AI/ML systems or LLM-based applications Preferred * Experience testing RAG pipelines or agentic workflows * Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.) * Exposure to: + Shift-left testing practices + Production observability and monitoring + Chaos or resilience testing Senior-Level Differentiators * Owned end-to-end AI test strategy and architecture * Defined quality metrics and release gates * Delivered scalable validation pipelines for production AI systems * Supported audit and compliance readiness Nice-to-have: * ISTQB certification * Cloud/ML certifications (AWS, Azure, GCP) * AI testing certifications ## Description We are looking for a highly motivated AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment., * Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows * Develop comprehensive test suites, including: + Unit, integration, and end-to-end (E2E) + Functional, regression, performance, and safety testing * Validate AI system behavior, including: + Non-deterministic LLM outputs + Hallucinations and edge cases + Multi-step agent decision-making * Design and manage evaluation systems: + Golden datasets + Benchmarking pipelines (accuracy, latency, reliability) * Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations * Implement observability and telemetry to enable traceability, monitoring, and audit readiness * Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria * Track and report quality KPIs, including test coverage, defect leakage, and system reliability * Drive root-cause analysis and continuous improvement across the AI testing lifecycle ## Related Videos - [pytest: Simple, rapid and fun testing with Python](https://www.wearedevelopers.com/videos/213-pytest-simple-rapid-and-fun-testing-with-python) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) - [Automagic Configuration in Python](https://www.wearedevelopers.com/videos/363-automagic-configuration-in-python) - [Hiring AI Native Talents](https://www.wearedevelopers.com/videos/100268-hiring-ai-native-talents) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)