Observability and Evaluation Engineer (AI/ML)

We are seeking an Observability and Evaluation Engineer with strong AI/ML expertise to build, monitor, evaluate, and optimize AI-driven applications, LLMs, and Agentic AI systems. The ideal candidate will have hands-on experience in observability, telemetry, monitoring, distributed systems, and AI model evaluation frameworks., * Design and implement observability frameworks for AI/ML and distributed applications.

• Build telemetry, logging, monitoring, and tracing solutions using OpenTelemetry and industry-standard tools.
• Develop evaluation pipelines for LLMs, AI agents, and RAG systems.
• Measure model accuracy, latency, reliability, hallucination rates, and business KPIs.
• Create dashboards, alerts, and AI system health monitoring solutions.
• Analyze production telemetry and model performance data to identify optimization opportunities.
• Establish AI quality metrics, evaluation standards, and benchmarking frameworks.
• Collaborate with AI Engineers, Data Scientists, Platform Teams, and SREs to improve system reliability and model effectiveness.

• 7+ years of experience in Software Engineering, SRE, MLOps, DevOps, or Observability Engineering.
• Strong experience in Observability, Monitoring, Telemetry, Logging, and Distributed Tracing.
• Hands-on expertise with OpenTelemetry, Grafana, Prometheus, Splunk, Datadog, Dynatrace, or New Relic.
• Experience building and monitoring AI/ML applications, LLMs, and RAG-based systems.
• Strong programming skills in Python and familiarity with Java or Golang.
• Experience with LLM evaluation frameworks, prompt testing, model performance analysis, and quality metrics.
• Knowledge of Vector Databases, embeddings, and Retrieval-Augmented Generation (RAG).
• Experience with AWS, Azure, or Google Cloud Platform cloud platforms.
• Strong understanding of Kubernetes, Docker, CI/CD, and MLOps practices.

<>Preferred Skills

• Experience with LangChain, LangGraph, LlamaIndex, MLflow, Arize, TruLens, LangSmith, or Weights & Biases.
• Knowledge of Agentic AI architectures and evaluation methodologies.
• Familiarity with AI safety, hallucination detection, and model guardrails.
• Experience with Kafka, Elasticsearch, and event-driven architectures.
• Understanding of SLI/SLO implementation for AI systems., * Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Science, or a related field.

<>Keywords

Observability, Evaluation Engineering, AI/ML, LLM, Agentic AI, RAG, OpenTelemetry, Monitoring, Telemetry, Logging, Distributed Tracing, LangChain, LangGraph, LangSmith, TruLens, MLflow, Arize, Python, Kubernetes, Docker, AWS, Azure, Google Cloud Platform, MLOps, SRE, Prometheus, Grafana, Datadog, Splunk.