> Markdown version of [/jobs/ext/3761835-director-data-science-quality-llm-judging-systems](https://www.wearedevelopers.com/jobs/ext/3761835-director-data-science-quality-llm-judging-systems). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Director, Data Science - Quality & LLM Judging Systems - **Company:** NEXT COMMERCE LLC - **Location:** San Mateo, CA, United States - **Experience:** Experienced - **Salary:** $169,000.0 - $338,000.0 - **Contract:** Permanent contract - **Skills:** Adobe Analytics, Training Data, A/B Testing, Web Accessibility, Data Analysis, Application Frameworks, Python (Programming Language), Machine Learning, Tensorflow, Scala (Programming Language), Web Content Accessibility Guidelines, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Apache Spark, Model Validation, Agentic-AI, Scikit Learn, Information Technology - **Published:** October 15, 2026 - **Apply:** https://www.themuse.com/jobs/walmart/director-data-science-quality-llm-judging-systems-for-conversational-commerce-53fa70 ## About the Role * 8+ years of experience in data science or applied machine learning * 5+ years leading teams focused on model evaluation, experimentation, or NLP applications * Deep experience with large language models, including prompt engineering, structured evaluation, and response grading * Familiarity with both human annotation workflows and LLM-based evaluators * Strong understanding of metric design, statistical evaluation methods, and A/B testing * Ability to translate ambiguous quality goals into concrete, testable evaluation frameworks * Excellent communication and cross-functional collaboration skills Preferred Qualifications * Advanced degree in Computer Science, Machine Learning, or related field * Experience with conversational AI, tool-augmented agents, or retrieval-augmented generation * Knowledge of efficient LLM adaptation techniques such as distillation, LoRA, or instruction tuning * Familiarity with evaluating outputs where objective ground truth is undefined (e.g., personalization, summarization, recommendation) * Track record of influencing product quality through principled evaluation and measurement, Option 1: Bachelors degree in Statistics, Economics, Analytics, Mathematics, Computer Science, Information Technology or related field and 6 years' experience in an analytics related field. Option 2: Masters degree in Statistics, Economics, Analytics, Mathematics, Computer Science, Information Technology or related field and 4 years' experience in an analytics related field. Option 3: 8 years' experience in an analytics or related field Preferred Qualifications... Outlined below are the optional preferred qualifications for this position. If none are listed, there are no preferred qualifications. Data science, machine learning, optimization models, PhD in Machine Learning, Computer Science, Information Technology, Operations Research, Statistics, Applied Mathematics, Econometrics, Successful completion of one or more assessments in Python, Spark, Scala, or R, Supervisory experience, Using open source frameworks (for example, scikit learn, tensorflow, torch), We value candidates with a background in creating inclusive digital experiences, demonstrating knowledge in implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly. The ideal candidate would have knowledge of accessibility best practices and join us as we continue to create accessible products and services following Walmart's accessibility standards and guidelines for supporting an inclusive culture. ## Description Walmart's Next Gen Commerce team is building intelligent, agentic systems that transform how customers shop through conversation. As Director, Data Science - Quality & LLM Judging Systems for Conversational Commerce, you will lead a critical pillar under the Senior Director of Data Science - Agentic AI for Conversational Commerce. Your mission is to define how we measure the effectiveness of the conversational shopping agents and the tools it invokes, ensuring we evaluate quality with both rigor and scale. You will lead a team responsible for defining evaluation metrics, designing measurement methodologies, and executing cost-efficient evaluations. This includes combining traditional human-labeled approaches with advanced "LLM-as-a-judge" techniques. You will design prompt-based evaluation tasks, identify when human oversight is needed, and explore how to distill smaller LLMs to replicate human-like evaluation at scale., Beyond conversation quality, your scope includes evaluating the outputs of tools invoked by the agent, such as personalized recommendations, summary generation, or proactive suggestions, where traditional metric-based evaluations fall short and human judgment is required. This is a hands-on leadership role requiring sharp judgment, strong experimental thinking, and fluency in both LLM prompting and applied ML. You will work closely with modeling, product, and platform teams to ensure that measurement drives improvement, and that the agent's behaviors align with quality, safety, and relevance at every step. Responsibilities * Grow and lead a high-performing team of data scientists, fostering a culture of technical excellence, fast execution, and clear accountability * Define evaluation strategy and success metrics for the conversational shopping agent and its tool outputs * Develop scalable measurement methodologies combining human-labeled benchmarks, LLM-as-a-judge prompts, and automated pipelines * Design and iterate on prompts that enable LLMs to perform structured evaluation tasks with high agreement to human judgment * Explore cost-effective alternatives by generating synthetic training data and distilling smaller LLMs to perform specific judging tasks * Establish quality review loops and integrate feedback from evaluations into model and product development * Partner with engineering, and product teams to ensure metrics are well-instrumented and align with long-term objectives * Drive tooling and process development to support reliable, reproducible, and efficient evaluation at scale