AI Evaluation Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+5 more
Job description
Day-to-Day Responsibilities (key tasks and expectations for the role)
- Define evaluation strategies and success metrics for AI agents.
- Build comprehensive test scenarios, including edge and adversarial cases.
- Develop automated evaluation pipelines and conduct regression testing.
- Analyze agent traces and outputs to identify performance bottlenecks.
- Collaborate with AI engineering and product teams for continuous improvement.
Requirements
We are seeking a skilled professional to join our team as an AI Evaluation Engineer, responsible for defining evaluation strategies and success metrics for AI agents. The role involves building test scenarios, developing automated evaluation pipelines, and collaborating with cross-functional teams to improve agent performance. Required Skills & Qualifications (Must-have qualifications that candidates must meet to be considered)
- Strong understanding of Generative AI, LLMs, RAG, prompts, tool calling, and AI agents.
- Proficiency in Python and experience working with APIs, JSON, datasets, and evaluation frameworks.
- Ability to translate business expectations into measurable evaluation criteria.
- Experience with AI evaluation/observability tools such as LangSmith, Langfuse, Ragas, DeepEval, MLflow, or similar.
- Prior work experience at client or in client’s Industry.
Applicants must be able to work directly for Artech on W2. Preferred Skills & Qualifications (Nice-to-have skills but are not required)
- Experience with automated evaluation pipelines.
- Knowledge of regression testing and establishing quality thresholds.
- Familiarity with edge and adversarial case testing.
Benefits & conditions
- Competitive salary and benefits package.
- Opportunities for professional growth and development.
- Supportive and inclusive work environment.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…