AI & Data Quality Engineer

Join our advanced AI Quality Engineering team working at the intersection of large language models and critical enterprise technology. We deliver innovative testing and quality assurance for next-generation AI-driven systems used by premier financial services and regulated-industry clients. Your expertise will directly impact the reliability, compliance, and performance of AI solutions across complex business workflows., Develop & Implement AI Quality Assurance: Craft comprehensive test plans for AI/ML-powered features (LLM assistants, knowledge retrieval, workflow AI), spanning functional, end-to-end, regression, UAT, and non-functional domains.

• Execute Thorough Evaluations: Assess system accuracy, relevance, safety, response time, and operational integrity across realistic use cases and production scenarios.

• Construct Evaluation Assets: Create and evolve test cases, datasets, and benchmarks specific to prompt engineering, retrieval-augmented generation, and user interactions.

• Collaborative Definition of Success: Work with development, product, and business teams to define acceptance criteria, quality gates, and KPIs for AI solutions.

• Risk & Defect Management: Identify weaknesses (hallucinations, model drift, edge cases, workflow vulnerabilities), drive proactive risk mitigation, and document corrective actions.

• Automate for Scale: Develop and support automation pipelines for continuous testing, model monitoring, reporting, and CI/CD integration.

• Data & API Validation: Validate interconnected data flows, APIs, microservices, and business logic dependencies supporting AI-empowered business systems.

• Governance & Quality Reporting: Drive release readiness, maintain risk logs, test documentation, and foster best practices for AI quality and model lifecycle management.

• Operational Feedback & Improvement: Track user feedback, incident metrics, and production signals to guide iterative improvement.

• Institutionalize Best Practices: Champion standards, toolkits, and controls for scalable, trustworthy AI delivery in regulated environments.

Bachelor’s degree or higher in Computer Science, Software Engineering, Data Science, Information Systems, or equivalent technical discipline.

• Experience:
• 6+ years in software quality engineering, test automation, or related technical QA roles.
• Demonstrated experience in testing enterprise business applications, APIs, ETL/data pipelines, or workflow platforms.
• Hands-on evaluation of AI, ML, or NLP systems, covering validation of model outputs, behaviors, and business logic.
• Technical Skills:
• Understanding of quality engineering principles, defect analysis, root cause investigation, and metric-driven reporting.
• Experience with test automation frameworks, API testing tools, and integration with CI/CD pipelines.
• Coding or scripting proficiency (e.g., Python, SQL) for test and data validation automation.
• Collaboration:
• Proven ability to work cross-functionally and communicate technical quality issues with both technical and non-technical stakeholders.

Preferred Additional Experience

• Specialized AI Testing: Familiarity with LLM prompt testing, RAG pipelines, synthetic/golden dataset design, or AI output auditing.

• Responsible AI/Model Governance: Knowledge of model validation best practices, AI observability, fairness, risk assessment, privacy, or compliance in regulated sectors.

• Industry Knowledge: Background working in financial services, wealth management, consulting, or similarly regulated industries.

Core Competencies

• Analytical, detail-oriented, and systematic approach to testing
• Creative curiosity focused on AI/model behaviors and uncovering operational risk
• Clear, concise documentation and communication skills
• Strong sense of ownership, reliability, and ethical responsibility