> Markdown version of [/jobs/ext/3071808-qa-automation-engineer](https://www.wearedevelopers.com/jobs/ext/3071808-qa-automation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # QA Automation Engineer - **Company:** BigBear.ai, Inc. - **Location:** United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** JavaScript (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Automation of Tests, Microsoft Azure, Continuous Integration, Customer Data Management, Data Validation, Software Debugging, Document Retrieval, Github, Python (Programming Language), PostgreSQL, Parsing, SQL Databases, Test Data, TypeScript, AI Infrastructure, Microsoft Power Automate, Retrieval-Augmented Generation, Large Language Models, Model Validation, Zapier, Generative AI, Backend, Git, SC Clearance, Integration Tests, Kubernetes, Bug Reporting, Low Latency, Playwright, Low-code, Data Management, Virtual Agents, GPT, Api Management, Docker - **Published:** September 25, 2026 - **Apply:** https://www.dice.com/job-detail/5502a9ea-4965-4474-b5fc-d17912654f90 ## About the Role * 4+years of experience QA automation engineering experience * Demonstrated experience building and maintaining automated tests with Playwright, including fixtures, resilient locators, assertions, network handling, and trace-based debugging. * Strong coding ability in TypeScript or JavaScript, with experience writing maintainable test code and reviewing changes through Git. * Experience with API testing, CI/CD integration, test isolation, and diagnosing failures across browser, application, and backend boundaries. * Ability to reason about complex requirements, explore edge cases, and balance testing depth with delivery priorities. Clear written and verbal communication: explaining defects, uncertainty, tradeoffs, and release risk to technical and nontechnical stakeholders. Experience testing authentication, authorization, permissions, and separation of customer data. * Familiarity with validating Defense in Depth behavior: confirming that multiple layers of controls work together, without this being a dedicated DevSec role. * Strong troubleshooting and analytical skills, with the ability to work independently and as part of a team. * Ability to obtain a Department of Defense Secret clearance What we'd like you to have * Experience testing LLM applications, retrieval-augmented generation, or AI agents. * Experience with Python for API testing, test utilities, test data generation, or AI evaluation workflows. * Experience with enterprise or government platforms and auditable test evidence. Experience testing model gateways, MCP tools, tool-using systems, or workflow automation and no-code/low-code platforms (e.g., Power Automate, Zapier, Make, n8n). * Familiarity with flagship Generative AI provider APIs (Google Vertex AI, AWS Bedrock, Microsoft Azure OpenAI) and models (OpenAI GPT, Anthropic Claude, Google Gemini). * Experience with CI/CD pipelines (e.g., GitHub Actions), Docker, Kubernetes, and observability/monitoring tooling. * Experience with PostgreSQL and SQL for test data setup, teardown, and validation. * Knowledge of government compliance frameworks (FedRAMP, NIST AI RMF, CMMC 2.0). * Active DoD security clearance at the Secret level or above. ## Description * Translate requirements into testable outcomes. Understand product goals, customer use cases, and complex feature interactions; identify ambiguities and define acceptance criteria with Product and Engineering. * Own automated test coverage. Design, implement, and maintain Playwright tests covering critical user journeys, feature functionality, and regressions, supported by API and integration testing. * Develop a risk-based testing strategy. Prioritize coverage according to customer impact, feature dependencies, and the product roadmap; adapt deliberately as priorities change. Test conversational AI workflows. Validate streaming responses, conversation history, file uploads, document retrieval and citations, model selection, and tool execution, including interruptions, timeouts, and partial failures. * Evaluate AI response quality. Build representative evaluation datasets and scoring criteria for accuracy, grounding, instruction following, and appropriate handling of unsafe requests. Account for natural variation in model responses. * Make automation reliable and useful. Integrate tests into CI/CD, investigate flaky tests, maintain isolated test data, and provide actionable failure diagnostics. * Communicate release readiness. Report defects with reproducible evidence, customer impact, and severity; explain coverage gaps and residual risks before UAT and release. * Preserve decision history. Document expected behavior, approved changes, and the rationale behind testing decisions so the team can distinguish intended changes from regressions. * Cover platform and AI infrastructure surfaces. Extend automation across backend APIs, authentication flows, billing and token behavior, model routing, AI workflow execution, file parsing, MCP/tool execution, agentic harnesses, and passthrough APIs, including provider-facing API compatibility. * Control test cost and execution footprint. Design AI workflow coverage that minimizes unnecessary token usage, external provider calls, latency, and execution cost without sacrificing signal. * Automate security-sensitive validation. Build repeatable coverage for user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, safe error handling, and layered control behavior. * Apply AI-assisted testing tools responsibly. Use AI assistance to accelerate test design, generation, triage, and maintenance while critically reviewing generated tests, assertions, and proposed repairs against reliability, reviewability, and deterministic validation standards. * Scale the automation footprint. Maintain test infrastructure, fixtures, and test data management that must grow with an expanding product surface area and an active engineering team. ## Related Videos - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Livecoding with AI](https://www.wearedevelopers.com/videos/1201-livecoding-with-ai) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [AI as a Test Designer: Transforming Experience into Automated Testing](https://www.wearedevelopers.com/videos/1984-ai-as-a-test-designer-transforming-experience-into-automated-testing) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) - [Are Classical Automation Frameworks Dead? How AI Agents Are Transforming QA](https://www.wearedevelopers.com/videos/100243-are-classical-automation-frameworks-dead-how-ai-agents-are-transforming-qa) ## Related Articles - [13 AI Tools You Have to Try](https://www.wearedevelopers.com/magazine/219-13-ai-tools-you-have-to-try) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline) - [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix) - [13 AI Tools for Developers](https://www.wearedevelopers.com/magazine/302-13-ai-tools-for-developers) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)