> Markdown version of [/jobs/ext/2313342-senior-data-scientist-ai-evaluation-improvement](https://www.wearedevelopers.com/jobs/ext/2313342-senior-data-scientist-ai-evaluation-improvement). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Scientist (AI Evaluation & Improvement) - **Company:** EVOLENT GROUP LLC - **Location:** Cheyenne, WY, United States (Remote available) - **Experience:** Expert - **Salary:** $135,000.0 - $165,000.0 - **Contract:** Permanent contract - **Skills:** A/B Testing, Artificial Intelligence, Data Analysis, Clinical Data Repository, Encodings, Python (Programming Language), Machine Learning, SQL Databases, Datadog, Large Language Models, AI Platforms, Information Technology - **Published:** August 30, 2026 - **Apply:** https://dejobs.org/x/x/9106808D40DC4FA09FC5A4C20BBA78FF/job/ ## About the Role * Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field - or equivalent experience. * 5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production. * 2+ years of recent, hands-on experience evaluating and improving LLM-based systems: structured error analysis, prompt/configuration iteration, experiment design, metrics interpretation - as a practitioner, not only as a reviewer of others' work. * Demonstrated experience with LLM evaluation methods and tools - golden/regression sets, LLM-as-judge with validation, tracing and observability tooling. * Strong Python and solid data-analysis skills (SQL a plus); comfort computing and reasoning about metrics such as sensitivity, specificity, and PPV. * Hypothesis-driven working style: the instinct to isolate variables and prove a fix, rather than tweak and hope. * Strong written communication - findings and go/no-go evidence must be legible to engineers, clinicians, and leadership. Preferred Requirements * Healthcare experience: utilization management, prior authorization, clinical documentation, or clinical/claims data; comfort reading clinical guideline and medical-policy content. * Experience mentoring data scientists or leading small technical workstreams; interest in growing into people leadership as a function scales. * Experience working with clinical reviewers or other domain experts to convert expert judgment into labeled data and evaluation criteria. * Experience with LLM observability/telemetry stacks (OpenTelemetry-based tracing, Logfire, Langfuse, or similar). * Statistics or experimentation background (A/B testing, statistical significance, sample-size reasoning). * Master's degree in a quantitative field. To ensure a secure hiring process we have implemented several identity verification steps, including submission of a government issued photo ID. We conduct identity verification during interviews, and final interviews may require onsite attendance. All candidates must complete a comprehensive background check, in-person I-9 verification, and may be subject to drug screening prior to employment. The use of artificial intelligence tools during interviews is prohibited and monitored. Misrepresentation will result in immediate disqualification from consideration. Technical Requirements: We require that all employees have the following technical capability at their home: High speed internet over 10 Mbps and, specifically for all call center employees, the ability to plug in directly to the home internet router. ## Description * Own the diagnostic loop for LLM-based clinical services: take failure modes surfaced by clinical reviewers and product managers, form root-cause hypotheses (prompt design, context assembly, retrieval, guideline encoding, model behavior, upstream data), and design experiments that isolate the cause. * Test candidate fixes - prompt and configuration variants, context changes, model alternatives - and verify improvements with structured evaluations, not anecdotes; confirm fixes don't regress other behavior. * Own the evaluation roadmap and quality bar for the team's AI services: which metrics gate deployment, how golden sets and regression suites grow, and what "good enough to ship" means in evidence. * Use and extend the team's evaluation platform: build golden sets and regression suites, define metrics (accuracy, guideline adherence, grounding/faithfulness), run and interpret eval batteries. Fluency in using modern eval tooling matters; the platform exists - extending it thoughtfully is in scope, rebuilding it is not. * Partner with clinical reviewers (medical directors) to turn review findings into labeled evidence and executable evaluation criteria; partner with product managers to prioritize which failure modes matter most. * Mentor others in evaluation methods and grow the evals function as it scales, including readiness to take direct reports as the team expands. * Support the annual clinical-guidelines update cycle with regression evaluation as guidelines, prompts, and models change; ramp with the team's senior data scientists in Q4 2026. * Document the diagnostic playbook: failure taxonomies, experiment templates, variant history - a method others can run, not a private intuition. This is a practicing role - the diagnostic loop is the job, at every level of seniority. * Handle clinical data (including PHI) according to organizational security, privacy, and compliance requirements. ## Related Videos - [AI for decision-making in Tech Recruiting](https://www.wearedevelopers.com/videos/1074-ai-for-decision-making-in-tech-recruiting) - [Bringing the power of AI to your application.](https://www.wearedevelopers.com/videos/1010-bringing-the-power-of-ai-to-your-application) - [A Brief History of Data Storage](https://www.wearedevelopers.com/videos/974-a-brief-history-of-data-storage) - [Debugging in the Dark](https://www.wearedevelopers.com/videos/1658-debugging-in-the-dark) - [Beyond Autocomplete: Local AI Code Completion Demystified](https://www.wearedevelopers.com/videos/961-beyond-autocomplete-local-ai-code-completion-demystified) - [JSON and Beyond](https://www.wearedevelopers.com/videos/968-json-and-beyond) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts) - [Prompt Engineering is a Job of the Past](https://www.wearedevelopers.com/magazine/342-prompt-engineering-is-a-job-of-the-past) - [The State of WebDev AI 2025 Results: What Can We Learn?](https://www.wearedevelopers.com/magazine/581-the-state-of-webdev-ai-2025-results-what-can-we-learn)