> Markdown version of [/jobs/ext/4582828-ai-evaluation-engineer](https://www.wearedevelopers.com/jobs/ext/4582828-ai-evaluation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Evaluation Engineer - **Company:** Sedgwick - **Location:** Jacksonville, FL, United States (Remote available) - **Experience:** Starter - **Contract:** Permanent contract - **Skills:** AI Evaluation, Artificial Intelligence, Amazon Web Services, Automation of Tests, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Monitoring of Systems, Statistical Hypothesis Testing, Python (Programming Language), Machine Learning, Role-Based Access Control, Regression Testing, Secure Coding, Test Data, Data Logging, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Model Validation, Generative AI, Langfuse — LLM Observability and Analytics Platform, LangSmith, Information Technology, RAGAS (Retrieval Augmented Generation Assessment), Arize Phoenix, Data Management, Machine Learning Operations, DeepEval, Meteor, Software Version Control, Promptfoo - **Published:** November 5, 2026 - **Apply:** https://www.jacksonvillejobforce.com/job.asp?id=3496656102&tx=UT5349TTT&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54 ## About the Role Master's degree in a computer science or a quantitative discipline from an accredited college or university preferred., 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market)., * Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs * A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance * Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review). * Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics. * Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation. * Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement. * Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent). * Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services. * Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders. * Excellent oral and written communication skills, including presentation skills * Ability to work in a team environment * Ability to meet or exceed Performance Competencies Preferred * Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation). * Background in ML evaluation, search/relevance, NLP, or QA automation. * Experience with LLMOps, monitoring and observability, and production feedback loops. * Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments. * Experience with data annotation workflows and working with SMEs. * Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption. SUCCESS MEASURES (first 12 months) * Standard evaluation framework adopted by pods. * Automated quality gates in CI/CD for production-bound solutions. * Calibrated LLM judges with documented agreement against human labels. * Reduction in regressions reaching production and in time to validate model or prompt changes. WORK ENVIRONMENT When applicable and appropriate, consideration will be given to reasonable accommodations. Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines Physical: Computer keyboarding, travel as required Auditory/Visual: Hearing, vision and talking The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time. ## Description * Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release. * Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency. * Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety). * Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort. * Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds. * Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes. * Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts). * Support A/B and shadow testing for model, prompt, and architecture changes. * Design, validate, and maintain LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels. * Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs. * Track judge reliability over time and re-calibrate as models and use cases change. * Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches. * Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data. * Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing. * Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews. * Contribute to responsible AI standards and release-readiness criteria. * Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites. * Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends. * Evaluate and recommend eval tooling and platforms; document standards and best practices. ADDITIONAL FUNCTIONS and RESPONSIBILITIES * Performs other duties as assigned. * Travel as required.