> Markdown version of [/jobs/ext/2715153-data-scientist](https://www.wearedevelopers.com/jobs/ext/2715153-data-scientist). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Scientist - **Company:** Ellipsis Health, Inc. - **Location:** San Francisco, CA, United States - **Experience:** Expert - **Salary:** $180,000.0 - $230,000.0 - **Contract:** Permanent contract - **Skills:** A/B Testing, Artificial Intelligence, Amazon Web Services, Business Analytics Applications, Data Analysis, Microsoft Azure, Big Data, Data Governance, Statistical Hypothesis Testing, Python (Programming Language), Machine Learning, Natural Language Processing, NumPy, Google Cloud, Large Language Models, Pandas, Scikit Learn, Information Technology, Data Management, Machine Learning Operations, Data Pipelines, Databricks - **Published:** September 4, 2026 - **Apply:** https://startup.jobs/senior-data-scientist-ellipsis-health-9814896 ## About the Role * 5+ years of industry experience in data science, machine learning, or a related quantitative field. * Bachelor's or Master's degree in Computer Science, Data Science, Statistics, Mathematics, or a related quantitative field. * Strong proficiency in programming languages such as Python (with libraries like Pydantic, Pandas, NumPy, Scikit-learn). * Strong foundation in statistical analysis, experimental design, hypothesis testing, and A/B testing methodologies. * Experience working with large-scale datasets, data pipelines, and analytical tools to generate actionable insights. * Familiarity with evaluation metrics (precision, recall, agreement metrics like Cohen's Kappa) to calibrate judge models against human ground truth. * Demonstrated ability to independently own and drive projects from problem definition through execution and implementation.. * Excellent communication and interpersonal skills, with the ability to collaborate effectively with AI/ML Engineers, Product Managers, and other cross-functional stakeholders in a fast-paced environment. Bonus Points If You Have: * Experience with Natural Language Processing (NLP) techniques and understanding of LLM architectures and their evaluation challenges. Experience with Speech Technologies and their evaluation is a significant advantage. * Experience in a regulated industry (e.g., healthcare, finance). Familiarity with data governance principles, particularly concerning PII/PHI. * Experience with MLOps practices and deploying models to production. * Familiarity with Langfuse, cloud platforms (GCP, AWS, Azure) and data platforms such as Databricks. ## Description * Drive High-Impact Projects: Own data science and AI evaluation projects end-to-end-from defining the problem and developing the methodology to executing the analysis, communicating results, and driving implementation. * Drive Continuous Improvement of LLM-as-a-Judge: Build the data and evaluation infrastructure for a closed-loop AI quality system, including golden datasets, ground-truth benchmarks, judge calibration, error analysis, and production feedback. Continuously identify failure modes and translate insights into improvements to datasets, evaluation methodologies, LLM-as-a-Judge systems, and the underlying AI products. * Develop Evaluation Frameworks: Design and implement robust evaluation frameworks, methodologies, and metrics for LLM- and speech-based AI systems. Develop scalable approaches that help us measure quality and accelerate customer onboarding. * Build and Manage Evaluation Datasets: Design, curate, and maintain high-quality datasets for training, testing, benchmarking, and evaluating conversational AI systems and their components. * Create and Manage Data Pipelines and Dashboards: Build and maintain scalable data pipelines and intuitive dashboards that enable reliable data analysis,ongoing AI quality monitoring, and data-driven decision-making across the organization. * Collaborate and Innovate: Work closely with AI/ML Engineers and Product Managers to integrate your findings into product enhancements. You'll stay current with the latest advancements in AI evaluation, particularly in Large Language Models (LLMs) and speech systems, contributing to our overall technical strategy. * Mentor and Lead: Mentor junior data scientists, establish best practices, and raise the technical bar across the team. ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Advanced Typing in TypeScript](https://www.wearedevelopers.com/videos/496-advanced-typing-in-typescript) - [Vectorize all the things! Using linear algebra and NumPy to make your Python code lightning fast.](https://www.wearedevelopers.com/videos/562-vectorize-all-the-things-using-linear-algebra-and-numpy-to-make-your-python-code-lightning-fast) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Coffee with Developers - Maria Apazoglou](https://www.wearedevelopers.com/videos/1209-coffee-with-developers-maria-apazoglou) ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production) - [How to start an AI project for a good cause and boost your career](https://www.wearedevelopers.com/magazine/15-how-to-start-an-ai-project-for-a-good-cause-and-boost-your-career)