> Markdown version of [/jobs/ext/4412848-ai-test-engineer](https://www.wearedevelopers.com/jobs/ext/4412848-ai-test-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Test Engineer

- **Company:** Tandym Tech
- **Location:** Vienna, VA, United States (Remote available)
- **Experience:** Experienced
- **Contract:** Permanent contract
- **Skills:** Testing (Software), AI Evaluation, A/B Testing, Application Programming Interfaces (APIs), Artificial Intelligence, Software Applications, Automation of Tests, Information Systems, Continuous Integration, Data Validation, Distributed Systems, Revision Control Systems, Python (Programming Language), Machine Learning, OpenAI, Software Engineering, Systems Integration, Strategies of Testing, GitHub Copilot, Prompt Libraries, LangChain, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Generative AI, Agentic-AI, Git, AI Platforms, LangSmith, Information Technology, HuggingFace, RAGAS (Retrieval Augmented Generation Assessment), DeepEval, Azure AI, Microservices
- **Published:** October 6, 2026
- **Apply:** https://www.dice.com/job-detail/db9af2c5-7aaf-4eba-9eda-f8ae33804315

## About the Role

3+ years of experience in software testing, quality engineering, test automation, AI/ML testing, or a related discipline
*
Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field
*
Strong programming and scripting skills in Python
*
Experience designing and implementing automated test strategies for complex software applications
*
Hands-on experience building automated test frameworks and testing pipelines
*
Experience testing APIs, integrations, microservices, or distributed applications
*
Experience working with CI/CD environments and modern software delivery practices
*
Working knowledge of machine learning, generative AI, large language models, or AI-powered applications
*
Experience with data validation, test-result analysis, defect investigation, and root-cause analysis
*
Knowledge of modern software testing methodologies, quality engineering practices, and software development lifecycles
*
Knowledge of statistical analysis and evaluation techniques used to validate system performance
*
Experience with Git or comparable source control systems and collaborative development practices

Desired Skills:
*
Experience testing generative AI applications, LLM-based solutions, AI agents, or Retrieval-Augmented Generation systems
*
Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, or comparable tools
*
Experience with LLM observability, monitoring, tracing, and evaluation practices
*
Experience with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms and frameworks
*
Familiarity with prompt engineering, benchmark dataset creation, prompt libraries, and AI output validation techniques
*
Understanding of Responsible AI principles, AI governance, model risk management, and enterprise AI controls
*
Experience with performance, scalability, reliability, and resilience testing
*
Experience supporting Developer Experience, developer productivity, software engineering enablement, or internal developer platforms
*
Experience working within enterprise, financial services, regulated, or highly controlled technology environments
*
Experience developing quality dashboards, automated reporting, or AI evaluation analytics
*
Familiarity with model comparison, A/B testing, experimentation, and statistical evaluation methodologies
*
Strong curiosity about emerging AI technologies and a passion for building scalable quality engineering capabilities

## Description

Design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, LLM, RAG, and AI-enabled developer experience solutions
*
Create and execute automated test suites to validate AI model behavior, output quality, accuracy, reliability, consistency, and performance
*
Develop automated evaluation pipelines integrated with CI/CD workflows to support continuous quality validation
*
Perform functional, regression, integration, performance, scalability, and scenario-based testing of AI-enabled applications and services
*
Analyze AI-generated outputs, investigate defects, and document testing findings, metrics, and recommendations for stakeholders