AI Test Engineer

Full Stack Development Azure Machine Learning Artificial Intelligence Business Transformation Concept Drift Detection Infrastructure as Code (IaC) Python (Programming Language) Systems Development Life Cycle Machine Learning Model Training Generative Artificial Intelligence MLOps (Machine Learning Operations) Artificial Intelligence Infrastructure Application Programming Interface (API), The Sr. AI Test Engineer is responsible for designing, building, and running the tests that prove AI systems behave as intended in production. Working under the direction of the AI Test Lead Architect, this role translates testing methodology into working test suites, evaluation harnesses, and quality gates for deterministic and non-deterministic systems (ML models, GenAI, and LLM applications).

The role is cloud-native by design: AI workloads are tested where they run, requiring deep expertise in one major cloud platform-AWS (SageMaker, Bedrock), GCP (Vertex AI), or Azure (Azure ML, Azure OpenAI)-with quality embedded directly into CI/CD and MLOps pipelines. The engineer partners closely with data scientists, ML engineers, and product teams to shift quality left and catch model, data, and behavioral issues before they reach users.

While this is a senior individual-contributor role, the Sr. AI Test Engineer is expected to mentor other testers, set technical standards for AI quality, and act as a trusted technical voice in client-facing conversations.

Roles and Responsibilities

AI Testing & Evaluation

• Design and implement test strategies for deterministic and non-deterministic AI systems (ML models, GenAI, LLMs), focusing on probabilistic correctness rather than simple pass/fail assertions.

• Build and maintain evaluation harnesses covering offline (benchmark datasets, golden sets) and online (production monitoring, A/B) evaluation.

• Validate LLM and GenAI behavior-hallucination, groundedness, prompt robustness, toxicity, and prompt-injection resilience-using automated and human-in-the-loop methods.

• Test for model quality and risk across accuracy, drift, robustness, bias, fairness, and explainability.

• Collect and analyze model quality metrics including Precision, Recall, F1, and Confusion Matrix, and translate results into clear quality signals.

Cloud & Platform Testing (AWS, GCP, or Azure)

• Test AI/ML workloads deployed on your primary cloud platform-AWS (SageMaker, Bedrock), GCP (Vertex AI), or Azure (Azure ML, Azure OpenAI)-validating model endpoints, inference performance, and scaling behavior.

• Validate data pipelines, feature stores, and model artifacts for quality, lineage, and consistency across cloud environments.

• Conduct performance, load, and latency testing of model-serving endpoints and GenAI APIs under realistic and adversarial conditions.

• Apply cloud-native testing patterns and infrastructure-as-code to make AI test environments reproducible.

Automation, Accelerators & Tooling

• Build reusable automation frameworks for AI regression testing, GenAI prompt validation, dataset validation, and drift detection.

• Establish AI quality gates embedded in CI/CD and MLOps workflows so model and data quality is verified on every change.

• Develop and evolve AI testing accelerators across SDLC integration, automation, and runtime monitoring/observability.

• Implement automated reporting that surfaces model quality, drift, and risk indicators to engineering and delivery teams.

Collaboration, Delivery & Client Engagement

• Partner with data science, ML engineering, and product teams to embed quality early and continuously (shift-left).

• Apply AI testing approaches across Agile, Waterfall, and hybrid delivery models.

• Engage confidently with technical client stakeholders; support AI quality assessments, demos, and proofs of value.

• Mentor junior testers and set technical standards for AI quality within the delivery team., Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications, but final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools. Related Jobs QA Engineer TEKsystems Charlotte, NC*Remote CI/CD MLflow Tooling Advocacy Test Case Vertex AI Pipelines Operations Leadership Management, Python (Programming Language) Systems Development Life Cycle Machine Learning Model Training Generative Artificial Intelligence MLOps (Machine Learning Operations) Artificial Intelligence Infrastructure Application Programming Interface (API) +0

MLflow Tooling Advocacy Test Case Vertex AI Pipelines Operations Leadership Management Automation AI Testing API Testing Data Quality Data Science Azure OpenAI Quality Gate Self-Starter Communication Observability Data Modeling AWS SageMaker Risk Reduction Data Pipelines Responsible AI Test Automation Microsoft Azure Problem Solving Data Validation AI/ML Inference Computer Science Machine Learning Systems Thinking Confusion Matrix Agile Methodology Quality Assessment Business Valuation Technical Standard Regression Testing Shift-Left Testing Workflow Management Amazon Web Services Testing Methodology Software Engineering Test Data Generation Deterministic Methods Waterfall Methodology, Core Skills & Experience

• Hands-on experience testing AI/ML and GenAI systems, including evaluation of training and inference, drift, bias, and explainability.

• Strong test automation skills with a programming language commonly used in AI (Python strongly preferred).

• Demonstrated experience building test or evaluation frameworks for ML or LLM systems.

• Familiarity with collecting and analyzing Precision, Recall, F1 Score, and Confusion Matrix.

• Experience integrating automated tests and quality gates into CI/CD and MLOps pipelines.

Technical & Platform Expertise

• Deep, hands-on expertise in one major cloud platform-AWS, GCP, or Azure-and its AI/ML services (e.g., SageMaker and Bedrock; Vertex AI; or Azure ML and Azure OpenAI). Familiarity with a second cloud is a plus.

• Test automation frameworks and data validation strategies.

• Monitoring, observability, and AI system reporting.

• Shift-left testing and continuous quality engineering.

• Familiarity with AI evaluation tooling (e.g., DeepEval, Ragas, LangSmith/Langfuse, Evidently, MLflow) is a strong plus.

Communication & Collaboration

• Clear communication with both technical and non-technical audiences.

• Consultative mindset focused on outcomes, risk reduction, and business value.

• Comfortable working in open, dynamic, and collaborative team environments.

Other Skills and Traits

• Strong analytical, problem-solving, and systems-thinking abilities.

• Self-starter with a proactive, ownership-driven mindset.

• Passionate advocate for quality, trust, and responsible AI.

• Desire to continuously improve AI quality processes and practices.

Education and Experience

• Minimum 6 years of experience in Quality Engineering, Testing, or Software Engineering.

• Minimum 2-3 years of hands-on experience testing or evaluating AI/ML or GenAI systems.

• Experience working with cloud-deployed AI workloads on a major cloud platform (AWS, GCP, or Azure).

• Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).

Additional Skills & Qualifications

-knowledge of traditional testing technologies such as CI/CD pipelines, test case management, API testing, and UI test automation is considered a plus for candidates

-that advanced skills in AI test automation, including shift-right testing, LLM testing, adversarial testing, prompt robustness, and test data generation, are beneficial but not mandatory for the role, AI Testing API Testing Data Quality Data Science Azure OpenAI Quality Gate Self-Starter Communication Observability Data Modeling AWS SageMaker Risk Reduction Data Pipelines Responsible AI Test Automation Microsoft Azure Problem Solving Data Validation AI/ML Inference Computer Science Machine Learning Systems Thinking Confusion Matrix Agile Methodology Quality Assessment Business Valuation Technical Standard Regression Testing Shift-Left Testing Workflow Management Amazon Web Services Testing Methodology Software Engineering Test Data Generation Deterministic Methods Waterfall Methodology Cloud-Native Computing Full Stack Development

This is a Contract position based out of Charlotte, NC. Pay and Benefits

The pay range for this position is $75.00 - $85.00/hr.

Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to specific elections, plan, or program terms. If eligible, the benefits available for this temporary role may include the following:

• Medical, dental & vision
• Critical Illness, Accident, and Hospital
• 401(k) Retirement Plan - Pre-tax and Roth post-tax contributions available
• Life Insurance (Voluntary Life & AD&D for the employee and dependents)
• Short and long-term disability
• Health Spending Account (HSA)
• Transportation benefits
• Employee Assistance Program
• Time Off/Leave (PTO, Vacation or Sick Leave) Workplace Type

We're partners in transformation. We help clients activate ideas and solutions to take advantage of a new world of opportunity. We are a team of 80,000 strong, working with over 6,000 clients, including 80% of the Fortune 500, across North America, Europe and Asia. As an industry leader in Full-Stack Technology Services, Talent Services, and real-world application, we work with progressive leaders to drive change. That's the power of true partnership. TEKsystems is an Allegis Group company., We're a leading provider of business and technology services. We accelerate business transformation for our customers. Our expertise in strategy, design, execution and operations unlocks business value through a range of solutions. We're a team of 80,000 strong, working with over 6,000 customers, including 80% of the Fortune 500 across North America, Europe and Asia, who partner with us for our scale, full-stack capabilities and speed. We're strategic thinkers, hands-on collaborators, helping customers capitalize on change and master the momentum of technology. We're building tomorrow by delivering business outcomes and making positive impacts in our global communities. TEKsystems and TEKsystems Global Services are Allegis Group companies. Learn more at TEKsystems.com.