> Markdown version of [/jobs/ext/3800949-senior-data-scientist-ai-evaluation-improvement](https://www.wearedevelopers.com/jobs/ext/3800949-senior-data-scientist-ai-evaluation-improvement). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior Data Scientist (AI Evaluation & Improvement)

- **Company:** EVOLENT GROUP LLC
- **Location:** Denver, CO, United States (Remote available)
- **Experience:** Expert
- **Salary:** $135,000.0 - $165,000.0
- **Contract:** Permanent contract
- **Skills:** AI Evaluation, A/B Testing, Data Analysis, Clinical Data Repository, Encodings, Python (Programming Language), Machine Learning, SQL Databases, Datadog, Large Language Models, Generative AI, Langfuse — LLM Observability and Analytics Platform, AI Platforms, Information Technology, Evaluation of Large Language Models
- **Published:** October 17, 2026
- **Apply:** https://www.denverjobsite.com/job.asp?id=3454775021&tx=KP7575FFQ&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54

## About the Role

* Bachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field - or equivalent experience.
 * 5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production.
 * 2+ years of recent, hands-on experience evaluating and improving LLM-based systems: structured error analysis, prompt/configuration iteration, experiment design, metrics interpretation - as a practitioner, not only as a reviewer of others' work.
 * Demonstrated experience with LLM evaluation methods and tools - golden/regression sets, LLM-as-judge with validation, tracing and observability tooling.
 * Strong Python and solid data-analysis skills (SQL a plus); comfort computing and reasoning about metrics such as sensitivity, specificity, and PPV.
 * Hypothesis-driven working style: the instinct to isolate variables and prove a fix, rather than tweak and hope.
 * Strong written communication - findings and go/no-go evidence must be legible to engineers, clinicians, and leadership.

Preferred Requirements

 * Healthcare experience: utilization management, prior authorization, clinical documentation, or clinical/claims data; comfort reading clinical guideline and medical-policy content.
 * Experience mentoring data scientists or leading small technical workstreams; interest in growing into people leadership as a function scales.
 * Experience working with clinical reviewers or other domain experts to convert expert judgment into labeled data and evaluation criteria.
 * Experience with LLM observability/telemetry stacks (OpenTelemetry-based tracing, Logfire, Langfuse, or similar).
 * Statistics or experimentation background (A/B testing, statistical significance, sample-size reasoning).
 * Master's degree in a quantitative field., We require that all employees have the following technical capability at their home: High speed internet over 10 Mbps and, specifically for all call center employees, the ability to plug in directly to the home internet router.

## Description

* Own the diagnostic loop for LLM-based clinical services: take failure modes surfaced by clinical reviewers and product managers, form root-cause hypotheses (prompt design, context assembly, retrieval, guideline encoding, model behavior, upstream data), and design experiments that isolate the cause.
 * Test candidate fixes - prompt and configuration variants, context changes, model alternatives - and verify improvements with structured evaluations, not anecdotes; confirm fixes don't regress other behavior.
 * Own the evaluation roadmap and quality bar for the team's AI services: which metrics gate deployment, how golden sets and regression suites grow, and what "good enough to ship" means in evidence.
 * Use and extend the team's evaluation platform: build golden sets and regression suites, define metrics (accuracy, guideline adherence, grounding/faithfulness), run and interpret eval batteries. Fluency in using modern eval tooling matters; the platform exists - extending it thoughtfully is in scope, rebuilding it is not.
 * Partner with clinical reviewers (medical directors) to turn review findings into labeled evidence and executable evaluation criteria; partner with product managers to prioritize which failure modes matter most.
 * Mentor others in evaluation methods and grow the evals function as it scales, including readiness to take direct reports as the team expands.
 * Support the annual clinical-guidelines update cycle with regression evaluation as guidelines, prompts, and models change; ramp with the team's senior data scientists in Q4 2026.
 * Document the diagnostic playbook: failure taxonomies, experiment templates, variant history - a method others can run, not a private intuition. This is a practicing role - the diagnostic loop is the job, at every level of seniority.
 * Handle clinical data (including PHI) according to organizational security, privacy, and compliance requirements.