> Markdown version of [/jobs/ext/3843368-observability-and-evaluation-engineer-ai-ml](https://www.wearedevelopers.com/jobs/ext/3843368-observability-and-evaluation-engineer-ai-ml). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Observability and Evaluation Engineer (AI/ML)

- **Company:** Mphasis
- **Location:** Charlotte, NC, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** LangGraph Framework, Java (Programming Language), Artificial Intelligence, Amazon Web Services, Microsoft Azure, Computer Programming, Continuous Integration, DevOps, Distributed Systems, Elasticsearch, Python (Programming Language), Machine Learning, Reliability Engineering, Prometheus, Software Safety, Software Engineering, Datadog, Data Logging, Google Cloud, LangChain, Evaluation Pipelines, Retrieval-Augmented Generation, Large Language Models, Grafana, Hallucination Detection, Model Validation, Llamaindex, Reliability of Systems, Generative AI, Agentic-AI, Event Driven Architecture, LangSmith, Kubernetes, Information Technology, Apache Kafka, Machine Learning Operations, Virtual Agents, Evaluation of Large Language Models, Splunk, New Relic (SaaS), Dynatrace, Docker, Golang
- **Published:** October 20, 2026
- **Apply:** https://www.dice.com/job-detail/bbcdb420-dca2-4ff5-b91a-46a883699374

## About the Role

* 7+ years of experience in Software Engineering, SRE, MLOps, DevOps, or Observability Engineering.
 * Strong experience in Observability, Monitoring, Telemetry, Logging, and Distributed Tracing.
 * Hands-on expertise with OpenTelemetry, Grafana, Prometheus, Splunk, Datadog, Dynatrace, or New Relic.
 * Experience building and monitoring AI/ML applications, LLMs, and RAG-based systems.
 * Strong programming skills in Python and familiarity with Java or Golang.
 * Experience with LLM evaluation frameworks, prompt testing, model performance analysis, and quality metrics.
 * Knowledge of Vector Databases, embeddings, and Retrieval-Augmented Generation (RAG).
 * Experience with AWS, Azure, or Google Cloud Platform cloud platforms.
 * Strong understanding of Kubernetes, Docker, CI/CD, and MLOps practices.

<>Preferred Skills

 * Experience with LangChain, LangGraph, LlamaIndex, MLflow, Arize, TruLens, LangSmith, or Weights & Biases.
 * Knowledge of Agentic AI architectures and evaluation methodologies.
 * Familiarity with AI safety, hallucination detection, and model guardrails.
 * Experience with Kafka, Elasticsearch, and event-driven architectures.
 * Understanding of SLI/SLO implementation for AI systems., * Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, or a related field.

<>Keywords

Observability, Evaluation Engineering, AI/ML, LLM, Agentic AI, RAG, OpenTelemetry, Monitoring, Telemetry, Logging, Distributed Tracing, LangChain, LangGraph, LangSmith, TruLens, MLflow, Arize, Python, Kubernetes, Docker, AWS, Azure, Google Cloud Platform, MLOps, SRE, Prometheus, Grafana, Datadog, Splunk.

## Description

We are seeking an Observability and Evaluation Engineer with strong AI/ML expertise to build, monitor, evaluate, and optimize AI-driven applications, LLMs, and Agentic AI systems. The ideal candidate will have hands-on experience in observability, telemetry, monitoring, distributed systems, and AI model evaluation frameworks., * Design and implement observability frameworks for AI/ML and distributed applications.

 * Build telemetry, logging, monitoring, and tracing solutions using OpenTelemetry and industry-standard tools.
 * Develop evaluation pipelines for LLMs, AI agents, and RAG systems.
 * Measure model accuracy, latency, reliability, hallucination rates, and business KPIs.
 * Create dashboards, alerts, and AI system health monitoring solutions.
 * Analyze production telemetry and model performance data to identify optimization opportunities.
 * Establish AI quality metrics, evaluation standards, and benchmarking frameworks.
 * Collaborate with AI Engineers, Data Scientists, Platform Teams, and SREs to improve system reliability and model effectiveness.