> Markdown version of [/jobs/ext/3876870-lead-software-development-engineer-in-test](https://www.wearedevelopers.com/jobs/ext/3876870-lead-software-development-engineer-in-test). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Software Development Engineer in Test

- **Company:** Alteryx, Inc.
- **Location:** Irvine, CA, United States (Remote available)
- **Experience:** Expert
- **Salary:** $165,000.0 - $185,000.0
- **Contract:** Permanent contract
- **Skills:** JavaScript (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Data Analysis, Automation of Tests, Continuous Integration, Software Debugging, Software Design Documents, Python (Programming Language), Load Testing, Selenium, Strategies of Testing, TypeScript, Large Language Models, Agentic-AI, Pytest, LangSmith, Playwright, Tools for Reporting, SDET, Alteryx
- **Published:** October 22, 2026
- **Apply:** https://www.disabledperson.com/jobs/76142373-lead-software-development-engineer-in-test

## About the Role

*
Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
*
Working knowledge of agent frameworks, preferably LangChain, with the ability to understand and debug context handling, tool execution, and agent state.
*
Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons across agent changes.
*
Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost.
*
Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems. JavaScript or TypeScript experience is a nice-to-have.
*
Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise; modeling experience is a plus.
*
Experience developing evaluation datasets and scoring methods, combining deterministic checks, human feedback, and LLM-based judges.
*
Ability to contribute at a technical lead level, guiding evaluation practices, mentoring engineers, integrating evaluations into CI/CD, and translating production failures into regression coverag

## Description

This Lead SDET will work on the test infrastructure that powers agent evaluation and observability for Ask Alteryx, partnering with engineering, product, and data science to measure and improve agent quality.

 * Analyze requirements to produce a comprehensive test strategy and detailed test cases.
 * Design and implement manual and automated tests at all levels in the test pyramid, including unit, component, integration, and end-to-end levels.
 * Design, build, and enhance scalable automated test frameworks to support feature testing and AI-driven product workflows.
 * Develop and maintain automation for UI and API tests using tools such as Selenium, pytest, and Playwright.
 * Conduct performance and load testing (using tools like locust) to ensure product scalability.
 * Build test reports using reporting tools such as Allure.
 * Report regressions to a corresponding dev team.
 * Influence design decisions by providing timely feedback on design documents and testability.
*
Collaborate closely with other members of an agile team, including engineers, product management, and UX, to become a valued member of an autonomous, cross-functional team.