> Markdown version of [/jobs/ext/3952584-machine-learning-evaluator-fully-remote](https://www.wearedevelopers.com/jobs/ext/3952584-machine-learning-evaluator-fully-remote). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Machine Learning Evaluator - Fully Remote

- **Company:** Mercor, Inc.
- **Location:** San Francisco, CA, United States (Remote available)
- **Experience:** Experienced
- **Salary:** $145,600.0 - $187,200.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Machine Learning, Tensorflow, Pytorch, Kaggle, Scikit Learn, Xgboost
- **Published:** October 27, 2026
- **Apply:** https://www.careerbuilder.com/job-details/machine-learning-evaluator-fully-remote-san-francisco-ca--39a6ae0b-de70-4bab-89be-bee6c2d6fd5a

## About the Role

Machine Learning, Artificial Intelligence (AI), Benchmarking, Research Laboratory, Auditing, Machine Learning, Experiment Design, Performance Modeling, Performance Management, Time Management, Must-Have

 * 3+ years hands-on applied/experimental ML experience.
 * Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene.
 * Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost).
 * Ability to critique ML claims against evidence and reproduce results.

Preferred

 * Competition/benchmark experience (e.g., Kaggle).
 * Graduate research or publication record in applied ML.
 * Prior task-grading or peer-review experience.

## Description

* Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks for training and evaluating models.
 * Assess experiment design, model-selection reasoning, and evaluation methodology.
 * Provide clear, rubric-based written feedback to improve AI model performance.
 * Review and critique ML claims against evidence and reproduce results.
 * Work independently and asynchronously to meet deadlines while enhancing model outputs.