Senior AI QA Automation Engineer
Encora Inc
Bolivia, NC, United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours
Job source
Tech stack
Application Programming Interfaces (APIs)
Artificial Intelligence
Automation of Tests
Microsoft Azure
Software Quality
Information Systems
Continuous Delivery
Continuous Integration
Fuzz Testing
Python (Programming Language)
Reliability Engineering
Selenium
+10 more
Software Engineering
Scripting
Retrieval-Augmented Generation
Multi-Agent Systems
Generative AI
Agentic-AI
Information Technology
Playwright
Prompt Injection
Evaluation of Large Language Models
Job description
- Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
- Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
- Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
- Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.
Requirements
Automation Engineering, Test Automation, Artificial Intelligence (AI), Selenium, Python Programming/Scripting Language, Testing, Test Automation, Application Programming Interface (API), Artificial Intelligence (AI), Microsoft Windows Azure, * 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
- Strong Python skills with experience building reusable automated testing frameworks.
- Hands-on experience with Playwright and/or Selenium.
- Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
- Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
- Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
- Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
- Experience implementing CI/CD quality gates and production monitoring.
- Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.
- Bachelorβs degree in Computer Science, Software Engineering, Information Systems, or a related field.
Preferred Skills:
- Experience testing Generative AI, RAG solutions, copilots, or AI agents.
- Experience with LLM evaluation frameworks and advanced prompt testing.
- Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.
- Experience with adversarial testing and Responsible AI testing.
- Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦