AI Data Quality Specialist position(Remote)

BA or BS degree +1-3 years of support and/or customer service or content design QA experience Preferred: financial institution or other highly regulated environment preferred Ability to evaluate conversations from a variety of angles and identify specific areas of improvement Ability to spend remain focused for long-running, repetitive tasks (eg labeling many datasets consistently) Ability to collaborate with AI scientist, product managers, and developers to understand project requirements and objectives. Able to evaluate a member’s experience by ensuring that we are providing the correct responses to our members with an empathetic and unbiased approach Design and create a diverse range of evaluators (LLM and rules-based), rubrics, and prompts tailored to specific AI tasks and applications. Develop creative and innovative ways to generate effective prompts that improve model performance. Conduct extensive research to identify and curate relevant data sources for rubric, prompt, and evaluator development. Ensure the judges, rubrics, and prompts are designed to elicit desired responses and achieve the desired outcomes. Optimize and fine-tune judges, rubrics, and prompts based on continuous testing and feedback from AI models, Agent outcomes, AI Scientists, Product Managers, developers. Analyze AI model outputs and make data-driven adjustments to improve prompt efficacy. Stay up-to-date with the latest advancements in natural language processing and AI technologies. Effective communicator in both verbal and written communication Excellent organization, detail oriented, and high level of accuracy Proven experience in resolving complex customer issues and managing escalations with internal and external teams Analytical skills with experience using Microsoft Excel or Google Sheets, SQL, python Applying data filters, creating charts and graphs, generating pivot tables, simple statistical analyses Experience with CRM and QA platforms

Thanks & regards