Data Scientist - AI Evaluation Analytics (West Coast Preferred)

• Support the development and refinement of AI evaluation metrics, scoring approaches, and benchmark methodologies.
• Validate datasets, data-quality criteria, evaluation inputs, and analytical assumptions to improve the reliability and reproducibility of results.
• Analyze evaluation data, performance trends, product telemetry, and measurement signals to generate actionable insights.
• Apply statistical analysis and experimentation techniques to assess AI system effectiveness and compare models, prompts, tools, or configurations.
• Identify trends, patterns, performance gaps, and opportunities to improve AI quality, workflow efficiency, user adoption, and customer outcomes.
• Communicate analytical findings, limitations, and recommendations clearly to product, engineering, and business stakeholders.

• Minimum of five years of professional experience in data science, advanced analytics, applied statistics, or a related field.
• At least two years of experience focused on AI evaluation analytics, generative AI measurement, machine learning evaluation, or AI performance analysis.
• Experience designing metrics, analytical frameworks, experiments, benchmarks, or measurement approaches.
• Proficiency in Python, SQL, and commonly used data-analysis and statistical libraries.
• Experience working with large datasets, product telemetry, operational data, or customer-usage data.
• Familiarity with generative AI, large language models, AI agents, coding assistants, or enterprise AI applications.
• Familiarity with tools and platforms such as OpenAI models and APIs, Anthropic Claude, Claude Code, Claude Opus, GitHub Copilot, or comparable AI platforms.
• Understanding of statistical methods such as hypothesis testing, regression analysis, confidence intervals, sampling, variance analysis, and experiment design.
• Ability to determine whether evaluation results are valid, reliable, reproducible, and meaningful.
• Strong communication skills with the ability to explain analytical results clearly.

Preferred Experience

• Experience evaluating AI/ML systems, generative AI applications, or agentic AI workflows.
• Experience with experimentation, benchmarking, model comparison, or performance analysis.
• Experience analyzing enterprise SaaS or PaaS applications, workflow platforms, ServiceNow applications, or similar enterprise technologies.
• Experience with software engineering, developer productivity, service-management, customer-support, or quality metrics.
• Familiarity with AI evaluation or observability tools such as OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval, or similar tools.
• Experience with dashboards and visualization platforms such as Tableau, Power BI, Looker, or Grafana.
• Familiarity with predictive analytics, forecasting, segmentation, anomaly detection, or causal inference when relevant to improving product and customer outcomes., * LinkedIn profile and work experience validation is mandatory. Please provide your LinkedIn Profile ID:
• This is a remote role; however, client prefers candidates who are located in West Coast. Work during the PDT hours is required. Can you work during PDT hours?
• Do you have hands on experience building, or evaluating AI-powered coding tools and software engineering agents, such as Claude Code, Devin, OpenCode, or comparable platforms?

Education:

• Bachelor’s (Required)

Experience:

• designing metrics and analytical frameworks: 3 years (Required)
• large datasets and extracting meaningful insights: 3 years (Required)
• background in data science, analytics, statistics: 5 years (Required)
• evaluating AI/ML systems and agentic AI applications: 2 years (Required)
• experimentation, benchmarking, or performance analysis: 2 years (Required)
• OpenAI Evals, LangSmith, MLflow, Ragas, DeepEval: 3 years (Preferred)

Language:

• English (Required)

$50 - $75 an hour - Temporary, Contract

Progile Solutions is seeking a Data Scientist - AI Evaluation Analytics for an exclusive client engagement supporting a global consulting leader and a Fortune 100 enterprise AI Platform-as-a-Service provider.

Provide data science and analytical expertise to improve the measurement, reliability, and interpretation of AI system performance through data-driven evaluation approaches. This role will support AI capabilities used within enterprise PaaS, workflow, and ServiceNow-related applications.