> Markdown version of [/jobs/ext/3956406-ai-evaluation-engineer](https://www.wearedevelopers.com/jobs/ext/3956406-ai-evaluation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Evaluation Engineer - **Company:** Artech Information Systems LLC - **Location:** Tampa, FL, United States - **Salary:** $79,040 - $87,360 - **Contract:** Temporary contract - **Skills:** AI Evaluation, Artificial Intelligence, JSON, Python (Programming Language), Regression Testing, Evaluation Pipelines, Retrieval-Augmented Generation, Large Language Models, Grafana, Generative AI, Langfuse — LLM Observability and Analytics Platform, Agentic-AI, LangSmith, RAGAS (Retrieval Augmented Generation Assessment), Machine Learning Operations, DeepEval, Api Design - **Published:** October 27, 2026 - **Apply:** https://www.careerjet.com/job/uscd201058e1a1abd5f42a0840faccb86e/eaa ## About the Role We are seeking a skilled professional to join our team as an AI Evaluation Engineer, responsible for defining evaluation strategies and success metrics for AI agents. The role involves building test scenarios, developing automated evaluation pipelines, and collaborating with cross-functional teams to improve agent performance. Required Skills & Qualifications (Must-have qualifications that candidates must meet to be considered) * Strong understanding of Generative AI, LLMs, RAG, prompts, tool calling, and AI agents. * Proficiency in Python and experience working with APIs, JSON, datasets, and evaluation frameworks. * Ability to translate business expectations into measurable evaluation criteria. * Experience with AI evaluation/observability tools such as LangSmith, Langfuse, Ragas, DeepEval, MLflow, or similar. * Prior work experience at client or in client's Industry. Applicants must be able to work directly for Artech on W2. Preferred Skills & Qualifications (Nice-to-have skills but are not required) * Experience with automated evaluation pipelines. * Knowledge of regression testing and establishing quality thresholds. * Familiarity with edge and adversarial case testing. ## Description Day-to-Day Responsibilities (key tasks and expectations for the role) * Define evaluation strategies and success metrics for AI agents. * Build comprehensive test scenarios, including edge and adversarial cases. * Develop automated evaluation pipelines and conduct regression testing. * Analyze agent traces and outputs to identify performance bottlenecks. * Collaborate with AI engineering and product teams for continuous improvement.