> Markdown version of [/jobs/ext/2705537-agentic-qa-engineer-generative-ai-agentic-systems-agent-multiagent-testing](https://www.wearedevelopers.com/jobs/ext/2705537-agentic-qa-engineer-generative-ai-agentic-systems-agent-multiagent-testing). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Agentic QA Engineer Generative AI & Agentic Systems (Agent, MultiAgent Testing) - **Company:** SolveIT Services Inc - **Location:** Dallas, TX, United States - **Experience:** Experienced - **Salary:** $124,800.0 - **Contract:** Permanent contract - **Skills:** Testing (Software), Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Microsoft Azure, Computer Programming, Continuous Integration, Distributed Systems, Failover, Github, Python (Programming Language), Systems Development Life Cycle, Queueing Systems, Reliability Engineering, Datadog, Large Language Models, Multi-Agent Systems, Event Driven Architecture, Containerization, Kubernetes, Low Latency, Machine Learning Operations, Virtual Agents, Software Version Control, Data Pipelines, Serverless Computing - **Published:** September 4, 2026 - **Apply:** https://www.dice.com/job-detail/71b26593-b4c5-4b11-aa2a-9bb801d6bfa8 ## About the Role * 7+ years in Software QA/Testing, with 2+ years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures. * Strong programming skills in Python experience building test harnesses, simulators, and fixtures. * Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails, and prompt testing. * Expertise in distributed systems testing latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering, and message queues. * Familiarity with orchestration frameworks (LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar). * Proficiency with CI/CD (GitHub Actions/Azure DevOps), observability (OpenTelemetry, PrometheGrafana, Datadog), and feature flags/canaries. * Solid understanding of privacy/security/compliance in AI systems (PII handling, content policies, model safety). * Excellent communication and leadership skills; proven ability to work cross-functionally with Ops, Data, and Engineering., * Experience with multi-agent simulators, agent graph testing, and tooling latency emulation. * Knowledge of MLOps (model versioning, datasets, evaluation pipelines) and A/B experimentation for LLMs. * Background in cloud (AWS), serverless, containerization, and event-driven architectures. Prior ownership of cost/latency/SLAs for AI workloads in production. ## Description We are seeking a hands-on AI Engineer to design and execute end-to-end testing strategies for agentic AI solutions, including multi-agent systems in production-grade environments. This role partners with the Agentic Operations Team to ensure resiliency, reliability, accuracy, latency, orchestration correctness, and scale. You will establish QA frameworks, build reusable test artifacts, drive macro-level validations across complex workflows, and lead the QA function for Agentic AI from Dev to Prod., * Agentic & MultiAgent Testing * Reliability, Resiliency, and Latency * Accuracy & Macro-Level Validations * Scale & Orchestration * Dev Prod Readiness * Define and own the QA strategy for agentic/multi-agent AI systems across dev, staging, and prod. * Mentor a team of QA engineers; establish testing standards, coding guidelines for test harnesses, and review practices. * Partner with Agentic Operations, Data Science, MLOps, and Platform teams to embed QA in the SDLC and incident response. * Design tests for agent orchestration, tool calling, planner-executor loops, and inter-agent coordination (e.g., task decomposition, handoff integrity, and convergence to goals). * Validate state management, context windows, memory/knowledge stores, and prompt/graph correctness under varying conditions. * Implement scenario fuzzing (e.g., adversarial inputs, prompt perturbations, tool latency spikes, degraded APIs). * Create resilience testing suites: chaos experiments, failover, retries/backoff, circuit-breaking, and degraded mode behavior. * Establish latency SLOs and measure end-to-end response times across orchestration layers (LLM calls, tool invocations, queues). * Ensure reliability through soak tests, canary verifications, and automated rollbacks. * Define ground-truth and reference pipelines for task accuracy (exact match, semantic similarity, factuality checks). * Build macro validation frameworks that validate task outcomes across multi-step agent workflows (e.g., complex data pipelines, content generation + verification agent loops). * Instrument guardrail validations (toxicity, PII, hallucination, policy compliance). * Design load/stress tests for multi-agent graphs under scale (concurrency, throughput, queue depth, backpressure). * Validate orchestrator correctness (DAG execution, retries, branching, timeouts, compensation paths). * Engineer reusable test artifacts (scenario configs, synthetic datasets, prompt libraries, agent graph fixtures, simulators). * Integrate tests into CI/CD (pre-merge gates, nightly, canary) and production monitoring with alerting tied to KPIs. * Define release criteria and run operational readiness (performance, security, compliance, cost/latency budgets). ## Related Videos - [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) - [System Resilience: Surviving the Software Storm](https://www.wearedevelopers.com/videos/874-system-resilience-surviving-the-software-storm) - [Debugging in the Dark](https://www.wearedevelopers.com/videos/1658-debugging-in-the-dark) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases](https://www.wearedevelopers.com/videos/1146-fault-tolerance-and-consistency-at-scale-harnessing-the-power-of-distributed-sql-databases) - [Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure](https://www.wearedevelopers.com/videos/1583-software-engineering-social-connection-yubo-s-lean-approach-to-scaling-an-80m-user-infrastructure) ## Related Articles - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path) - [A 5-Step Open-Source Setup for Agentic Engineering](https://www.wearedevelopers.com/magazine/738-a-5-step-open-source-setup-for-agentic-engineering) - [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)