Automation Test Engineer

• Experience testing AI-powered applications, including LLM prompt and model regression testing.
• Experience testing non-deterministic conversational flows, with assertions that tolerate variation without hiding real regressions.
• Experience generating synthetic test data at scale.
• Built test automation frameworks from the ground up.
• Python or TypeScript skills, with CI/CD experience such as GitHub Actions or Jenkins.
• Track record of raising quality standards across a team.

• Ownership of test strategy and infrastructure for a cutting-edge AI platform.
• Top-tier health coverage: medical, dental, and vision.
• 401(k).
• Paid time off, holidays, and paid parental leave.

Compensation

• $150,000

What You’ll Be Doing

• Design, build, and maintain automated test infrastructure covering unit, integration, end-to-end, and load testing.
• Build evaluation and regression harnesses that score AI behavior by outcome, not exact output.
• Generate synthetic conversation data and de-identified patient scenarios to cover rare edge cases.
• Build testing frameworks and tooling that help the wider engineering team write reliable tests.
• Define and enforce quality standards and best practices across engineering teams.
• Own test environments and CI/CD integration for automated test runs.
• Triage production quality issues and drive root cause analysis, including HIPAA and compliance-aware testing.

Our client is a healthcare technology company where software quality directly affects whether patients get the care and access they need. You’ll own the testing strategy and automation for an AI-driven platform, making sure every behavior and integration works as expected before it ever reaches a live patient. It’s a rare chance to solve testing problems where the system never gives the same answer twice.