> Markdown version of [/jobs/ext/2722672-member-of-technical-staff-data-scientist](https://www.wearedevelopers.com/jobs/ext/2722672-member-of-technical-staff-data-scientist). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Member of Technical Staff (Data Scientist - **Company:** ONE STOP COLLECTIBLE CORP - **Location:** San Francisco, CA, United States - **Contract:** Permanent contract - **Skills:** Amazon Web Services, Python (Programming Language), Machine Learning, SQL Databases, Large Language Models, Production Code, Databricks - **Published:** September 4, 2026 - **Apply:** https://startup.jobs/member-of-technical-staff-data-scientist-evals-perplexity-7805766 ## About the Role * PhD or MS in a technical field or equivalent experience * 4+ years of experience in data science or machine learning * Strong proficiency in Python and SQL (expected to write production-grade code) * Experience building within a modern cloud data stack, specifically AWS and Databricks * Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster, * 1+ years of experience working with LLMs at scale, specifically with LLM-as-a-judge setups * Prior experience working on customer-facing web products or consumer apps, with real user traffic at scale * A strong research background, with experience applying research methods to real-world ML problems * Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets ## Description * Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness * Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality * Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices * Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements * Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality ## Related Videos - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Designing the Future of Human<>Agent Collaboration](https://www.wearedevelopers.com/videos/1447-designing-the-future-of-human-agent-collaboration) - [Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases](https://www.wearedevelopers.com/videos/1146-fault-tolerance-and-consistency-at-scale-harnessing-the-power-of-distributed-sql-databases) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) ## Related Articles - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)