Senior ML / AI Engineer

• LLM QA - the core technology behind Toloka’s automated quality-check mechanism. Every annotation flowing through Self-Service is reviewed by an LLM agent we design, train, and operate.
• Model distillation and fine-tuning - adapting frontier and open-source models to Toloka’s tasks to hit the right quality at the right cost.
• Evaluation, benchmarking, cost modeling, and model selection across providers.

We own the full chain. The same team designs the ML solution, ships it to production, keeps it running 24/7, analyzes the results coming back from real projects, and feeds that signal into the next iteration. No hand-off between research, engineering, and operations - it’s all us.

About the Position

As a Senior Software Engineer, you will develop core product functionality, improving AI agent capabilities, and bringing new AI features to production. You will work closely with the ML team to extend agent infrastructure, evolve hybrid human-expert workflows, and design robust observability metrics to ensure top-tier system reliability.

What you’ll do

• Extend the agent’s capabilities and add new product features to improve user experience - from clarifying task requirements to working with final results.

• Improve collaboration between the agent and human experts, evolving hybrid workflows and balancing quality and speed.

• Contribute to our agent infrastructure: tools, environments, internal knowledge, and third-party integrations.

• Design and implement observability metrics for agent quality and overall system reliability.

• Develop and maintain production services in Python, * Personal and career development: we support employee’s ambition for professional development and encourage them to implement their ideas in their projects
• Flexibility: we offer a remote or hybrid employment. You will also design with your manager a workday that works best for you
• Paid parental leave and sick leave
• Medical insurance fully covered for the employee + 80% of coverage for family members
• Meal allowance, transportation expenses reimbursement
• 25 vacation days per year
• Tax and migration support if necessary

• Extensive background (5+ years) in software engineering, featuring advanced proficiency in Python programming;
• Proven track record of at least 2 years building and launching autonomous AI agents;
• Deep enthusiasm for artificial intelligence and a drive to apply disciplined engineering standards to machine learning;
• Competency in architecting high-scale backend systems and refining software delivery workflows;
• Fluency in English at a B2 level or above.

Nice to have

• Hands-on experience utilizing Java or C++;
• Strong foundation in machine learning acquired through rigorous academic research or substantial industry practice;
• Demonstrated success in deploying sophisticated LLM-based applications within production environments;
• Practical knowledge of orchestration technologies and diverse agentic development frameworks.

About Toloka

At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains-from doctors and lawyers to physicists and engineers-and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages. We are a well-funded startup with an enviable portfolio of clients including Anthropic, Amazon, Microsoft, Poolside, Recraft, and Shopify.

Recently, we secured strategic investment led by Bezos Expeditions and Nebius Group with participation from Mikhail Parakhin, CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA, UK, the Netherlands, Czech Republic, Serbia, and more.

About the Team

We are the ML team inside Toloka - we build the machine-learning products that power the platform itself, so every project running on Toloka is faster, cheaper, and more reliable.

A few examples of what we own