> Markdown version of [/jobs/ext/3998159-senior-ai-evaluation-engineer](https://www.wearedevelopers.com/jobs/ext/3998159-senior-ai-evaluation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior AI Evaluation Engineer

- **Company:** LTM Inc
- **Location:** United States (Remote available)
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** LangGraph Framework, AI Evaluation, Java (Programming Language), A/B Testing, Amazon Web Services, Amazon S3, COBOL (Programming Language), Continuous Integration, Data Control, IBM DB2, Identity and Access Management, Spring Framework, Python (Programming Language), Regression Testing, Datadog, Data Logging, Evaluation Pipelines, Retrieval-Augmented Generation, Large Language Models, Sybase, AgentCore, Agentic-AI, CrewAI, Functional Programming, Cloudwatch
- **Published:** October 28, 2026
- **Apply:** https://www.dice.com/job-detail/e5451c59-9351-4eac-87f3-05c1edfae2f2

## About the Role

· GenAI Experience (Level 400) with Advanced Python

· Experience building production grade systems

· Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring

· LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity

· Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety

· Agentic framework experience (Strands, LangGraph, CrewAI or similar)

· AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have

· Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)

· Agent-as-a-Judge framework; HITL evaluation pipelines

· Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction

· RAG quality evaluation (faithfulness, context precision, recall)

· CI/CD integration for eval regression; observability tooling (tracing, structured logging)

· Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design

· Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)

· Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

## Description

· Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)

· Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows

· Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety

· Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices