> Markdown version of [/jobs/ext/2068142-ai-automation-qa-engineer](https://www.wearedevelopers.com/jobs/ext/2068142-ai-automation-qa-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Automation QA Engineer

- **Company:** Unisoft Technology Inc
- **Location:** Baltimore, MD, United States
- **Contract:** Permanent contract
- **Skills:** Agile Methodology, Artificial Intelligence, Automation of Tests, Information Systems, Python (Programming Language), Machine Learning, Natural Language Processing, Selenium, Software Engineering, Software Quality Assurance (SQA), Test Data, Strategies of Testing, Postman, Large Language Models, Pytest, Integration Tests, Performance Monitor, Data Pipelines
- **Published:** August 15, 2026
- **Apply:** https://www.dice.com/job-detail/d9276329-b99e-4ac7-a2a0-5de5af9caf63

## About the Role

General Experience: The proposed candidate must have at least eight (8) years of information systems quality assurance experience. 2-5+ years of experience in software quality assurance, with at least 1-2 years specifically in AI/ML or automation testing.

Technical Skills: Proficiency in Python (essential for GenAI libraries like LangChain and LlamaIndex) and API testing tools (Postman, REST).

Domain Knowledge: Solid understanding of machine learning concepts, data pipelines, and NLP (natural language processing).

Drive software quality assurance lifecycle within an Agile process

Establish and coordinate test strategies with development/product teams

Design and implement test plans and test cases

Develop and execute automated UI and functional tests

Enhance and maintain automated CI flows

## Description

We are looking for a AI Automation QA Engineer who are local to DMV area. This position requried 100% Onsite and In-Person Interview in Baltimore,MD.

Responsibilities:

AI/LLM Testing: Validating AI outputs, including GenAI hallucination detection, prompt-response accuracy, and evaluating models using metrics like BLEU or perplexity.

Test Automation: Developing and maintaining automated test suites for AI systems, including integrating tools like PyTest, Selenium, or specialized AI frameworks such as RAGAS and promptfoo into CI/CD pipelines.

Collaboration: Working closely with data scientists, developers, and product managers to define, test, and implement AI features within Agile environments.

Performance Monitoring: Monitoring AI system metrics such as latency, cost, and model drift

Develop and document test cases, testing plans and procedures in an agile environment.

Develop, execute and coordinate IT software tests and evaluate results to ensure compliance with applicable regulations.

Design and prepare all test data needed. Reviews test results and evaluates for conformance to design.

Develop and document test cases, testing plans and procedures in an agile environment.

Develop and maintain automated regression and integration test plans for validation.


## Related Videos

- [pytest: Simple, rapid and fun testing with Python](https://www.wearedevelopers.com/videos/213-pytest-simple-rapid-and-fun-testing-with-python)
- [Are Your APIs Ready for AI Agents](https://www.wearedevelopers.com/videos/2004-are-your-apis-ready-for-ai-agents)
- [Proactive monitoring and smoke testing in your production environment](https://www.wearedevelopers.com/videos/139-proactive-monitoring-and-smoke-testing-in-your-production-environment)
- [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing)
- [Old tools, new tricks](https://www.wearedevelopers.com/videos/1916-old-tools-new-tricks)
- [Hiring AI Native Talents](https://www.wearedevelopers.com/videos/100268-hiring-ai-native-talents)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts)
- [Top AI Tools for Developers in 2025](https://www.wearedevelopers.com/magazine/560-top-ai-tools-for-developers-in-2025)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Transforming Software Development: The Role of AI and Developer Tools](https://www.wearedevelopers.com/magazine/527-transforming-software-development-the-role-of-ai-and-developer-tools)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)