Software Engineer in Test

We are seeking a Mid to Senior Software Development Engineer in Test (SDET) to join our engineering team and help build and evolve a robust test infrastructure for a sophisticated Conversational AI platform supporting voice and chat channels.

This is not a traditional QA role. We are looking for an engineer with a strong software development and automation background who can design and build reusable test frameworks, test harnesses, and automation solutions. The SDET will work closely with Product, Architecture, and Engineering teams to establish testing strategies, improve test coverage, and enable developers to take greater ownership of quality.

The ideal candidate will have strong programming skills in Java or Node.js, extensive API and automation experience, and exposure to Conversational AI, voice/chat applications, AI agents, or LLM testing., * Understand the overall software ecosystem and develop testing approaches aligned with the platform architecture and engineering strategy.

• Collaborate with Product, Architecture, and Engineering teams to define test strategies, coverage objectives, acceptance criteria, and quality standards.
• Design and develop test strategies and reusable test harnesses for Conversational AI systems across Voice and Chat channels.
• Build, maintain, and enhance reusable automation frameworks that enable engineers to test efficiently.
• Establish clear patterns and best practices for test framework selection, test data setup, automation implementation, and test execution.
• Develop comprehensive examples, documentation, and reusable components that allow developers to write automated tests without reinventing solutions.
• Provide test infrastructure that enables engineering teams to independently test and own the quality of their services.
• Develop and execute automated tests across Java/Node.js backend services, REST APIs, and frontend applications.
• Build specialized testing strategies for conversational AI workflows and AI agent interactions.
• Develop test harnesses for API-level, backend service, integration, and end-to-end testing.
• Validate complex multi-turn conversations, intent recognition, context handling, and orchestration logic.
• Develop and automate Voice and Chat test plans.
• Test AI-generated responses for accuracy, consistency, relevance, and expected behavior.
• Apply modern approaches to AI/LLM evaluation, including LLM-as-a-Judge and related evaluation techniques.
• Identify testing gaps and recommend practical solutions to improve product quality and reliability.
• Partner with developers on complex testing scenarios and contribute to automation and development activities.
• Monitor test execution, quality metrics, defect trends, and overall test coverage.
• Support release quality by ensuring defects are identified, tracked, and resolved within expected SLAs.
• Validate that platform changes maintain or improve the overall customer experience.
• Educate and mentor engineers on test automation frameworks, testing patterns, and best practices.
• Promote test-driven development and automation practices across engineering teams.
• Remove friction from the testing process by making test infrastructure and automation easy for developers to use.
• Contribute to automated test implementation and general software development activities when test infrastructure initiatives are complete.
• Help identify and close testing gaps across engineering teams.

• 5+ years of total software engineering experience, with at least 3+ years focused on SDET, test automation, or software engineering in test.
• Strong hands-on experience developing automated testing solutions, frameworks, and test infrastructure.
• Strong programming experience in at least one backend language:
• Java
• Node.js / JavaScript
• Strong understanding of Object-Oriented Design (OOD) and software engineering principles.
• Extensive experience with REST API testing, including:
• Request/response validation
• API automation
• API mocking
• Contract testing
• Experience with test frameworks such as:
• JUnit
• Jest
• or equivalent frameworks
• Experience with automation and testing tools such as:
• Playwright
• Selenium
• Cucumber
• Postman
• SOAPUI
• Bruno
• Experience building or maintaining custom test frameworks and test harnesses.
• Hands-on experience with Voice and Chat test automation.
• Familiarity with testing Conversational AI, chatbots, voice bots, NLP-based systems, or AI agents.
• Experience testing conversational workflows involving multiple turns, context, intent recognition, and orchestration.
• Experience with CI/CD pipelines, preferably GitHub Actions.
• Strong understanding of test data management and experience developing effective test data strategies.
• Comfortable performing full-stack testing across APIs, backend services, and frontend applications.
• Experience testing AI/LLM-based applications and evaluating AI-generated responses.
• Familiarity with techniques such as LLM-as-a-Judge, LLM evaluation, or AI response validation.
• Strong communication and documentation skills.
• Ability to collaborate effectively with software engineers, architects, product teams, and other stakeholders.
• Ability to mentor engineers and promote automation and testing best practices.
• Must be available to work onsite in Hartford, CT or Wellesley, MA, or be willing to relocate.

Preferred Qualifications

• Deep hands-on experience developing automation frameworks, testing tools, and APIs.
• Experience designing test infrastructure used by multiple engineering teams.
• Professional experience with Conversational AI platforms.
• Experience testing AI agents and agentic workflows.
• Experience with Cyara and/or Hammer.
• Strong knowledge of API mocking and contract testing tools.
• Experience with performance, load, or scalability testing.
• Experience integrating automated testing into CI/CD pipelines.
• Experience working in cloud-based software development environments.
• Experience with Test-Driven Development (TDD) and modern software engineering practices.

What Success Looks Like

The successful SDET will help create a testing ecosystem where developers can easily build, execute, and maintain automated tests. You will establish reusable frameworks and patterns, improve coverage across APIs, backend services, Voice, Chat, and AI workflows, and help engineering teams deliver reliable features faster.

This position is ideal for a software engineer who specializes in testing and automation-not a traditional manual QA or test execution role.