> Markdown version of [/jobs/ext/4667717-machine-learning-engineer](https://www.wearedevelopers.com/jobs/ext/4667717-machine-learning-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Machine Learning Engineer

- **Company:** AI + AI, LLC
- **Location:** New York, NY, United States
- **Experience:** Experienced
- **Salary:** $200,000 - $300,000
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Software Debugging, Machine Learning, Enterprise Messaging Systems, Performance Tuning, Regression Testing, Speech Recognition, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Deep Learning, Agentic-AI, Information Technology, Evaluation of Large Language Models, Data Pipelines, Data Generation
- **Published:** November 11, 2026
- **Apply:** https://www.dice.com/job-detail/c06bc669-d41a-4892-b5ea-2c525983b957

## About the Role

Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.

8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.

Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.

Hands-on experience with failure analysis and LLM evaluation.

Experience building automated evaluation systems.

Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.

Practical experience training or fine-tuning voice and speech models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.

Preferred Qualifications

Master's or PhD in Computer Science, Machine Learning, Statistics, or a related field.

Experience evaluating LLMs or agentic systems, including LLM-as-a-judge and RAG evaluation.

Experience with synthetic data generation and prompt engineering.

Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, and low-latency inference optimization for production AI applications.

## Description

Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.

Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data. Continuously iterate on model architecture, data composition, and training strategies to improve accuracy, naturalness, and latency.

Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.

Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.

Develop annotation guidelines and workflows for human-labeled evaluation data. Calibrate LLM-as-judge systems against human reviewers to maintain reliable automated evaluations over time.

Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rates, and safety signals in production while identifying model or data drift before it impacts customers.

Own the metric standards for all published AI performance metrics, including definitions, populations, data granularity, rollups, and validity windows.

Establish release gates and offline regression testing suites that every AI feature must pass before changes to prompts, models, or configurations are deployed. Measure reliability across repeated trials rather than relying solely on average pass rates.

Develop voice-specific evaluation frameworks using simulated callers across accents, languages, background noise, interruptions, DTMF inputs, and tool failures, with latency and speech recognition accuracy as key quality metrics.