> Markdown version of [/jobs/ext/1208721-ai-test-engineer](https://www.wearedevelopers.com/jobs/ext/1208721-ai-test-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Test Engineer - **Company:** SCHROCK CONCRETE, INC. - **Location:** Baltimore, MD, United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Testing (Software), Application Programming Interfaces (APIs), Agile Methodology, Artificial Intelligence, Amazon Web Services, Automation of Tests, Big Data, Software as a Service, Cloud Computing, Software Quality, Data as a Services, Data Validation, Information Engineering, Database Queries, Distributed Systems, Machine Learning, Scrum Methodology, Software Safety, Software Engineering, Software Quality Assurance (SQA), Systems Integration, Strategies of Testing, Scripting, Delivery Pipeline, Large Language Models, Prompt Engineering, Model Validation, Information Technology, Virtual Agents, Data Pipelines, Databricks - **Published:** July 8, 2026 - **Apply:** https://www.careerjet.com/job/us1b5247b8832ea526bee59d7830ae1d49/eaa ## About the Role * Bachelor's degree in Computer Science, Data Science, Engineering, or a related technical discipline (or equivalent professional experience) * 3+ years of experience in software quality assurance, testing, or software engineering * Experience testing APIs, distributed systems, data pipelines, or cloud-based applications * Strong understanding of software testing methodologies, test planning, and quality engineering principles * Experience developing automated testing solutions * Strong SQL skills and experience validating complex datasets * Experience working within Agile software development environments * Familiarity with AI and machine learning concepts, including model evaluation, inference, prompt engineering, and model behavior * Experience working with cloud technologies such as AWS or Databricks * Strong analytical and troubleshooting skills with the ability to identify root causes of complex technical issues * Excellent written and verbal communication skills with the ability to collaborate across technical and business teams ## Description SG2 Recruiting is supporting Trend Health in the search for an AI Test Engineer II who will help ensure the quality, reliability, and safety of AI-powered capabilities across the organization's healthcare platform. This is not a traditional software QA position. You'll be responsible for validating machine learning models, large language model (LLM) outputs, AI-driven workflows, and intelligent automation that directly support healthcare operations. Working alongside engineering, data, AI, and product teams, you'll develop comprehensive testing strategies that ensure AI systems produce accurate, explainable, and reliable results while meeting business objectives and clinical expectations. If you're excited about combining software quality engineering with emerging AI technologies and want to help build trustworthy AI solutions in healthcare, we'd like to hear from you. What You Will Be Doing AI Testing & Validation * Designing comprehensive testing strategies for AI and machine learning models, including functional, regression, integration, and scenario-based testing * Executing manual and automated validation of AI-generated outputs, including summaries, classifications, recommendations, and predictive results * Developing test cases that evaluate edge cases, model drift, prompt behavior, and unexpected AI responses * Measuring model performance against defined business, technical, and quality success criteria * Documenting testing results and communicating findings to engineering and AI stakeholders AI Workflow & Agent Validation * Testing AI agent workflows, orchestration logic, automation pipelines, and multi-step AI processes * Validating integrations between AI agents, APIs, data services, and downstream applications * Identifying workflow failures, performance bottlenecks, and opportunities to improve reliability * Verifying end-to-end functionality across AI-enabled business processes Data Quality & Integrity * Validating training, validation, and testing datasets for completeness, consistency, quality, and potential bias * Performing data validation across ingestion and processing pipelines to ensure accurate inputs and outputs * Partnering with Data Engineering teams to identify anomalies, inconsistencies, and data quality issues Test Automation & Engineering * Building and maintaining automated testing frameworks for AI-enabled applications * Developing scripts and tooling that enable scalable, repeatable validation of AI-generated outputs * Integrating automated testing into CI/CD pipelines to support continuous validation and rapid releases * Improving testing efficiency through automation and reusable test assets AI Safety & Governance * Defining and validating AI guardrails, including accuracy thresholds, explainability standards, and fallback logic * Detecting risks associated with hallucinations, bias, model drift, and unreliable outputs * Collaborating with Engineering and AI teams to improve AI safety, reliability, and production readiness Collaboration & Continuous Improvement * Partnering with Engineering, Product, QA, AI, Data Engineering, and Operations teams throughout the software development lifecycle * Participating in Agile ceremonies, sprint planning, backlog refinement, and release validation activities * Contributing to testing standards, best practices, documentation, and quality processes * Supporting root cause analysis for AI-related production issues and implementing long-term quality improvements, MANTECH seeks a highly motivated and collaborative Software Engineer to join our team in Columbia, MD. In this role, you will develop, maintain, and enhance complex and diverse sof… + 12 days ago ## Related Videos - [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [JavaScript? No. Java Scripts! - Scripting with Java](https://www.wearedevelopers.com/videos/2094-javascript-no-java-scripts-scripting-with-java) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Back to the Roots: Testing in the Age of AI](https://www.wearedevelopers.com/videos/100021-back-to-the-roots-testing-in-the-age-of-ai) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) ## Related Articles - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [The 8 Best Code Testing Tools](https://www.wearedevelopers.com/magazine/402-the-8-best-code-testing-tools) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [What is Software Engineering in the Age of AI?](https://www.wearedevelopers.com/magazine/640-what-is-software-engineering-in-the-age-of-ai)