> Markdown version of [/jobs/ext/4659422-machine-learning-eval-engineer](https://www.wearedevelopers.com/jobs/ext/4659422-machine-learning-eval-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Machine Learning Eval Engineer

- **Company:** Recruiting from Scratch
- **Location:** San Francisco, CA, United States (Remote available)
- **Salary:** $150,000 - $300,000
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Amazon S3, Spreadsheets, Data Infrastructure, Software Debugging, Information Extraction, Python (Programming Language), Machine Learning, Language Modeling, Online Analytical Processing, Systems Development Life Cycle, Software Deployment, Software Engineering, TypeScript, Unstructured Data, Web Applications, Evaluation Pipelines, Flask (Web Framework), Large Language Models, Prompt Engineering, AI Platforms, Information Technology, Machine Learning Operations, Evaluation of Large Language Models, Intelligent Document Processing
- **Published:** November 10, 2026
- **Apply:** https://www.careerbuilder.com/job-details/machine-learning-eval-engineer-san-francisco-ca--a2ab0012-dc64-485e-aea7-09e1d52cbe6d

## About the Role

Machine Learning, Machine Learning, Artificial Intelligence (AI), Continuous Improvement, Quality Management, Customer Relations, Process Improvement, Production Systems, Scalable System Development, Benchmarking, * 1-5 years of experience in Machine Learning Engineering, Software Engineering, or ML Infrastructure

 * Strong sweet spot around 2-4 years of experience
 * Experience building evaluation systems, ML tooling, or data infrastructure from zero-to-one
 * Experience working at high-bar technology companies, AI startups, quantitative firms, or leading research organizations
 * Experience working with production LLM applications
 * Experience building customer-facing ML tooling or internal AI platforms
 * Startup experience strongly preferred
 * Demonstrated ownership of high-impact technical initiatives

Technical Requirements

 * Strong Python engineering skills
 * Deep understanding of LLM evaluation methodologies including LLM-as-a-Judge
 * Strong prompt engineering experience
 * Strong understanding of precision, recall, statistical evaluation, and ML metrics
 * Experience building evaluation pipelines or benchmarking systems
 * Comfortable building lightweight web applications using Flask, TypeScript, or similar frameworks
 * Experience working with unstructured data including documents, PDFs, OCR, or document extraction
 * Familiarity with AWS S3, OLAP systems, Tinybird, or analytics infrastructure preferred
 * Experience working with Vision-Language Models or document AI preferred
 * Strong debugging, experimentation, and software engineering fundamentals, * Bachelor's degree in Computer Science, Mathematics, Physics, Machine Learning, or related technical field preferred
 * Strong academic background from a top engineering or quantitative program preferred
 * Formal machine learning education or research experience preferred

Soft Skills

 * Strong analytical thinking
 * High ownership mentality
 * Excellent communication skills
 * Comfortable explaining technical concepts to non-technical stakeholders
 * Comfortable operating in ambiguity
 * Self-directed and proactive
 * Bias toward execution with technical precision
 * Startup mentality
 * Strong engineering craftsmanship

## Description

* Develop benchmarks, metrics, and automated evaluation pipelines measuring model quality
 * Build workflows that identify failure modes across large-scale unstructured datasets
 * Design statistical evaluation methodologies using precision, recall, and model quality metrics
 * Build internal tooling and lightweight applications for model visualization and evaluation analysis
 * Work hands-on with enterprise documents including PDFs, spreadsheets, OCR outputs, and unstructured data
 * Partner closely with ML engineers to prioritize model improvements using evaluation insights
 * Build customer-specific benchmarks demonstrating model performance across real-world workflows
 * Design evaluation infrastructure supporting production AI systems operating at massive scale
 * Collaborate with GTM, Product, and Engineering teams to communicate model performance
 * Prototype new evaluation techniques leveraging LLM-as-a-Judge methodologies
 * Own evaluation systems from initial design through production deployment