> Markdown version of [/jobs/ext/4158492-machine-learning-engineering-technical-leader](https://www.wearedevelopers.com/jobs/ext/4158492-machine-learning-engineering-technical-leader). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Machine Learning Engineering Technical Leader

- **Company:** Cisco Systems, Inc.
- **Location:** San Francisco, CA, United States
- **Experience:** Expert
- **Salary:** $216,300.0 - $280,800.0
- **Contract:** Permanent contract
- **Skills:** Computer Networks, Continuous Integration, Fuzz Testing, Monitoring of Systems, Python (Programming Language), Machine Learning, Tensorflow, System Testing, Management of Software Versions, Pytorch, Large Language Models, Deep Learning, Generative AI, Agentic-AI, Information Technology, Drift Detection, Data Generation
- **Published:** October 26, 2026
- **Apply:** https://www.sanfranciscogigs.com/job.asp?id=3474068886&tx=HT7973TTI&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54

## About the Role

* BA/BS with 8+ years of relevant experience, MS with 6+ years, or PhD with 3+ years in Computer Science, Machine Learning or a related quantitative field.
 * Experience taking machine learning or LLM-based systems into production and owning them operationally, not only building prototypes.
 * Experience in Python and modern deep learning frameworks such as PyTorch or TensorFlow.
 * Experience designing evaluation, measurement or quality systems for ML or LLM products - datasets, metrics, judges, benchmarks or regression suites.
 * Experience setting technical direction across multiple teams through influence rather than reporting lines., * LLM-as-judge design and calibration: practical experience building judge systems and validating them against human experts, including inter-annotator agreement and judge drift.
 * Agentic system evaluation: measuring multi-step reasoning, tool selection, trajectory quality and root-cause correctness rather than single-turn output quality.
 * Production monitoring at scale: asynchronous scoring of live traffic, drift detection, and alerting that engineering teams actually act on.
 * Simulation and synthetic environments: building reproducible test environments or synthetic data generation for domains where ground truth is scarce.
 * Evaluation in CI/CD: integrating automated evaluation into build pipelines and using it to gate releases.
 * Networking, observability or infrastructure domain knowledge: familiarity with the telemetry, topology and failure modes of enterprise network and observability systems.
 * Cost-aware inference: routing, model selection and optimisation techniques that keep evaluation economically viable at production volume.
 * Technical community contribution: publications at top ML venues, open-source contributions, or public technical writing in the evaluation space.

## Description

* Own the evaluation roadmap and strategy for agentic AI quality, from architecture to production.
 * Design a layered evaluation architecture: deterministic checks on every PR, model-based evaluation pre-release, and human expert calibration.
 * Build and calibrate multi-model LLM-as-judge panels, tracking true-positive and true-negative rates separately.
 * Run the human-in-the-loop flywheel with SMEs, turning annotations into training signal for automated judges.
 * Score production traffic online, alert on regressions and drift, and convert real traces into offline test cases.
 * Lead simulation, synthetic data, and adversarial testing without consuming lab resources.
 * Close the loop from evaluation failure to clustered issue, candidate fix, and shadow/A/B validation.
 * Ensure trustworthy results through experiment snapshots, dataset versioning, coverage reporting, and cost tracking.
 * Drive common standards, data contracts, and judge criteria across partner teams without direct authority.
 * Mentor engineers, set evaluation engineering standards, and represent the domain to senior leadership.