Machine Learning Engineer

Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.

Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data. Continuously iterate on model architecture, data composition, and training strategies to improve accuracy, naturalness, and latency.

Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.

Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.

Develop annotation guidelines and workflows for human-labeled evaluation data. Calibrate LLM-as-judge systems against human reviewers to maintain reliable automated evaluations over time.

Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rates, and safety signals in production while identifying model or data drift before it impacts customers.

Own the metric standards for all published AI performance metrics, including definitions, populations, data granularity, rollups, and validity windows.

Establish release gates and offline regression testing suites that every AI feature must pass before changes to prompts, models, or configurations are deployed. Measure reliability across repeated trials rather than relying solely on average pass rates.

Develop voice-specific evaluation frameworks using simulated callers across accents, languages, background noise, interruptions, DTMF inputs, and tool failures, with latency and speech recognition accuracy as key quality metrics.

Bachelor’s degree in Computer Science, Machine Learning, Statistics, or a related field.

8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.

Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.

Hands-on experience with failure analysis and LLM evaluation.

Experience building automated evaluation systems.

Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.

Practical experience training or fine-tuning voice and speech models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.

Preferred Qualifications

Master’s or PhD in Computer Science, Machine Learning, Statistics, or a related field.

Experience evaluating LLMs or agentic systems, including LLM-as-a-judge and RAG evaluation.

Experience with synthetic data generation and prompt engineering.

Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, and low-latency inference optimization for production AI applications.

Salary: $200,000 - $300,000 per year, $181,000 - $250,000 USD

Why join us?

Key moment to join Aircall in terms of growth and opportunities

? Our people matter, work-life balance is important at Aircall

Fast-learning environment, entrepreneurial and strong team spirit

45+ Nationalities: cosmopolite & multi-cultural mindset

Competitive salary package & benefits

DE&I Statement:

At Aircall, we believe diversity, equity and inclusion - irrespective of origins, identity, background and orientations - are core to our journey.

We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive.

We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.

Why join us?

An exciting time to join a fast-growing company with significant opportunities for growth and career development.

? A people-first culture that prioritizes employee well-being and work-life balance.

A fast-paced, entrepreneurial environment with continuous learning and a strong team-oriented culture.

A diverse, international team representing more than 45 nationalities and perspectives.

Competitive compensation package and comprehensive benefits.

We are a fast-growing, AI-powered communications platform transforming how businesses connect with their customers.

For over a decade, we’ve believed voice builds the strongest business relationships. Our platform brings voice, SMS, WhatsApp, and more than 250 integrations together in one seamless workspace, helping businesses drive revenue, resolve customer issues faster, and build lasting relationships. Our AI-powered voice agents handle routine interactions, allowing teams to focus on the conversations that matter most.

Today, our platform is trusted by more than 24,000 businesses worldwide.

We’re 600+ people from 40+ nationalities across Paris, London, Madrid, Berlin, New York, San Francisco, Seattle, Sydney, and Mexico City. We’re ambitious, product-driven, and execution-focused-built for visible impact, fast decisions, and real growth.

Come join us!

Aircall’s AI suite includes an AI Voice Agent and AI Messaging Agent that autonomously handle calls, WhatsApp, and SMS, plus AI Assist, which delivers real-time coaching, call summaries, and CRM automation for sales and support teams. We are looking for someone that can build out the evaluation foundation across all of these products and other agentic products. You’ll work on voice models, agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products. You will set up repeatable pipelines for regression testing and benchmarking as models and features evolve so teams can ship confidently without re-inventing evaluation methodology for each product.