AI Automation QA Engineer

Unisoft Technology Inc

Baltimore, MD, United States

4 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience required

1 year minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Agile Methodology Artificial Intelligence Automation of Tests Information Systems Python (Programming Language) Machine Learning Natural Language Processing Selenium Software Engineering Software Quality Assurance (SQA) Test Data Strategies of Testing

+6 more

Postman Large Language Models Pytest Integration Tests Performance Monitor Data Pipelines

Job description

We are looking for a AI Automation QA Engineer who are local to DMV area. This position requried 100% Onsite and In-Person Interview in Baltimore,MD.

Responsibilities:

AI/LLM Testing: Validating AI outputs, including GenAI hallucination detection, prompt-response accuracy, and evaluating models using metrics like BLEU or perplexity.

Test Automation: Developing and maintaining automated test suites for AI systems, including integrating tools like PyTest, Selenium, or specialized AI frameworks such as RAGAS and promptfoo into CI/CD pipelines.

Collaboration: Working closely with data scientists, developers, and product managers to define, test, and implement AI features within Agile environments.

Performance Monitoring: Monitoring AI system metrics such as latency, cost, and model drift

Develop and document test cases, testing plans and procedures in an agile environment.

Develop, execute and coordinate IT software tests and evaluate results to ensure compliance with applicable regulations.

Design and prepare all test data needed. Reviews test results and evaluates for conformance to design.

Develop and document test cases, testing plans and procedures in an agile environment.

Develop and maintain automated regression and integration test plans for validation.

Requirements

General Experience: The proposed candidate must have at least eight (8) years of information systems quality assurance experience. 2-5+ years of experience in software quality assurance, with at least 1-2 years specifically in AI/ML or automation testing.

Technical Skills: Proficiency in Python (essential for GenAI libraries like LangChain and LlamaIndex) and API testing tools (Postman, REST).

Domain Knowledge: Solid understanding of machine learning concepts, data pipelines, and NLP (natural language processing).

Drive software quality assurance lifecycle within an Agile process

Establish and coordinate test strategies with development/product teams

Design and implement test plans and test cases

Develop and execute automated UI and functional tests

Enhance and maintain automated CI flows