> Markdown version of [/jobs/ext/1738190-data-scientist-ai-evaluation-analytics-west-coast-preferred](https://www.wearedevelopers.com/jobs/ext/1738190-data-scientist-ai-evaluation-analytics-west-coast-preferred). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Scientist - AI Evaluation Analytics (West Coast Preferred) - **Company:** PROGILE SOLUTIONS, LLC - **Location:** United States (Remote available) - **Experience:** Experienced - **Salary:** $104,000.0 - $156,000.0 - **Contract:** Temporary contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Data Analysis, Analysis of Variance (ANOVA), Big Data, Statistical Hypothesis Testing, Python (Programming Language), Machine Learning, Regression Analysis, Operational Data Store, Platform as a Service (PAAS), Performance Tuning, Power BI, ServiceNow Application Engine, Software Engineering, SQL Databases, Tableau (Software), Enterprise Software Applications, GitHub Copilot, Large Language Models, Grafana, Generative AI, AI Platforms, Machine Learning Operations, OPUS (Software), Looker Analytics - **Published:** July 31, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=24f3b5c310a572e4 ## About the Role * Minimum of five years of professional experience in data science, advanced analytics, applied statistics, or a related field. * At least two years of experience focused on AI evaluation analytics, generative AI measurement, machine learning evaluation, or AI performance analysis. * Experience designing metrics, analytical frameworks, experiments, benchmarks, or measurement approaches. * Proficiency in Python, SQL, and commonly used data-analysis and statistical libraries. * Experience working with large datasets, product telemetry, operational data, or customer-usage data. * Familiarity with generative AI, large language models, AI agents, coding assistants, or enterprise AI applications. * Familiarity with tools and platforms such as OpenAI models and APIs, Anthropic Claude, Claude Code, Claude Opus, GitHub Copilot, or comparable AI platforms. * Understanding of statistical methods such as hypothesis testing, regression analysis, confidence intervals, sampling, variance analysis, and experiment design. * Ability to determine whether evaluation results are valid, reliable, reproducible, and meaningful. * Strong communication skills with the ability to explain analytical results clearly. Preferred Experience * Experience evaluating AI/ML systems, generative AI applications, or agentic AI workflows. * Experience with experimentation, benchmarking, model comparison, or performance analysis. * Experience analyzing enterprise SaaS or PaaS applications, workflow platforms, ServiceNow applications, or similar enterprise technologies. * Experience with software engineering, developer productivity, service-management, customer-support, or quality metrics. * Familiarity with AI evaluation or observability tools such as OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval, or similar tools. * Experience with dashboards and visualization platforms such as Tableau, Power BI, Looker, or Grafana. * Familiarity with predictive analytics, forecasting, segmentation, anomaly detection, or causal inference when relevant to improving product and customer outcomes., * LinkedIn profile and work experience validation is mandatory. Please provide your LinkedIn Profile ID: * This is a remote role; however, client prefers candidates who are located in West Coast. Work during the PDT hours is required. Can you work during PDT hours? * Do you have hands on experience building, or evaluating AI-powered coding tools and software engineering agents, such as Claude Code, Devin, OpenCode, or comparable platforms? Education: * Bachelor's (Required) Experience: * designing metrics and analytical frameworks: 3 years (Required) * large datasets and extracting meaningful insights: 3 years (Required) * background in data science, analytics, statistics: 5 years (Required) * evaluating AI/ML systems and agentic AI applications: 2 years (Required) * experimentation, benchmarking, or performance analysis: 2 years (Required) * OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval: 3 years (Preferred) Language: * English (Required) ## Description * Support the development and refinement of AI evaluation metrics, scoring approaches, and benchmark methodologies. * Validate datasets, data-quality criteria, evaluation inputs, and analytical assumptions to improve the reliability and reproducibility of results. * Analyze evaluation data, performance trends, product telemetry, and measurement signals to generate actionable insights. * Apply statistical analysis and experimentation techniques to assess AI system effectiveness and compare models, prompts, tools, or configurations. * Identify trends, patterns, performance gaps, and opportunities to improve AI quality, workflow efficiency, user adoption, and customer outcomes. * Communicate analytical findings, limitations, and recommendations clearly to product, engineering, and business stakeholders. ## Related Videos - [5 steps for running a Kubernetes environment at scale](https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Beyond Dashboards: Fixing Text-to-SQL with Semantic RAG](https://www.wearedevelopers.com/videos/2036-beyond-dashboards-fixing-text-to-sql-with-semantic-rag) - [Hiring AI Native Talents](https://www.wearedevelopers.com/videos/100268-hiring-ai-native-talents) - [Data Analytics with Microsoft Fabric: End-to-End Use Case with Data Agents](https://www.wearedevelopers.com/videos/1547-data-analytics-with-microsoft-fabric-end-to-end-use-case-with-data-agents) - [All your telemetry data from any source in one place](https://www.wearedevelopers.com/videos/57-all-your-telemetry-data-from-any-source-in-one-place) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [The State of WebDev AI 2025 Results: What Can We Learn?](https://www.wearedevelopers.com/magazine/581-the-state-of-webdev-ai-2025-results-what-can-we-learn) - [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this)