AI Evaluation Engineer

Artech Information Systems LLC
Tampa, FL, United States
22 days ago
Apply on www.careerjet.com
Prepare application

Role details

Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Compensation
$79,040 - $87,360
Working hours
Regular working hours

Tech stack

AI Evaluation Artificial Intelligence JSON Python (Programming Language) Regression Testing Evaluation Pipelines Retrieval-Augmented Generation Large Language Models Grafana Generative AI Langfuse — LLM Observability and Analytics Platform Agentic-AI
+5 more
LangSmith RAGAS (Retrieval Augmented Generation Assessment) Machine Learning Operations DeepEval Api Design

Job description

Day-to-Day Responsibilities (key tasks and expectations for the role)

  • Define evaluation strategies and success metrics for AI agents.
  • Build comprehensive test scenarios, including edge and adversarial cases.
  • Develop automated evaluation pipelines and conduct regression testing.
  • Analyze agent traces and outputs to identify performance bottlenecks.
  • Collaborate with AI engineering and product teams for continuous improvement.

Requirements

We are seeking a skilled professional to join our team as an AI Evaluation Engineer, responsible for defining evaluation strategies and success metrics for AI agents. The role involves building test scenarios, developing automated evaluation pipelines, and collaborating with cross-functional teams to improve agent performance. Required Skills & Qualifications (Must-have qualifications that candidates must meet to be considered)

  • Strong understanding of Generative AI, LLMs, RAG, prompts, tool calling, and AI agents.
  • Proficiency in Python and experience working with APIs, JSON, datasets, and evaluation frameworks.
  • Ability to translate business expectations into measurable evaluation criteria.
  • Experience with AI evaluation/observability tools such as LangSmith, Langfuse, Ragas, DeepEval, MLflow, or similar.
  • Prior work experience at client or in client’s Industry.

Applicants must be able to work directly for Artech on W2. Preferred Skills & Qualifications (Nice-to-have skills but are not required)

  • Experience with automated evaluation pipelines.
  • Knowledge of regression testing and establishing quality thresholds.
  • Familiarity with edge and adversarial case testing.

Benefits & conditions

  • Competitive salary and benefits package.
  • Opportunities for professional growth and development.
  • Supportive and inclusive work environment.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerjet.com
Prepare application

Good distractions

Loading talks and stories from around this role…