> Markdown version of [/jobs/ext/4008687-ai-monitoring-engineer](https://www.wearedevelopers.com/jobs/ext/4008687-ai-monitoring-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Monitoring Engineer - **Company:** Ova Technologies - **Location:** New York, NY, United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** AI Evaluation, Artificial Intelligence, Amazon Web Services, Data Analysis, Microsoft Azure, Bash Shell, Cloud Computing, Computer Programming, Computer Networks, Databases, Linux, DevOps, Distributed Systems, Fiddler (Software), Python (Programming Language), PostgreSQL, Linux System Administration, Machine Learning, Redis, Reliability Engineering, Prometheus, Standard Sql, Azure Machine Learning, SQL Databases, AI Infrastructure, Datadog, Data Logging, Scripting, Google Cloud, Large Language Models, Grafana, Hallucination Detection, Mttr, Generative AI, Agentic-AI, AI Platforms, Kubernetes, Information Technology, Machine Learning Operations, Drift Detection, Azure AI, Model Inference, Splunk, New Relic (SaaS), Dynatrace, Docker, Elk Stack - **Published:** October 29, 2026 - **Apply:** https://www.thejobnetwork.com/job/ac775b57-799d-47b2-95e6-229fc05916b0/ai-monitoring-engineer ## About the Role * Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field. * 3 years of experience in Monitoring Engineering, Site Reliability Engineering (SRE), DevOps, Platform Engineering, MLOps, or Cloud Operations. * Experience monitoring production AI or machine learning systems. * Strong programming skills in Python and scripting with Bash. * Experience with Linux administration. * Hands-on experience with Docker and Kubernetes. * Experience with AWS, Microsoft Azure, or Google Cloud Platform. * Strong understanding of distributed systems, cloud infrastructure, and networking. * Experience with observability tools such as Prometheus, Grafana, OpenTelemetry, ELK Stack, Datadog, Splunk, or New Relic. * Experience with SQL and time-series databases for operational reporting. Preferred Qualifications * Experience with machine learning monitoring platforms such as Evidently AI, Arize AI, Fiddler AI, WhyLabs, or Azure AI monitoring capabilities. * Experience monitoring Large Language Models (LLMs) and Generative AI applications. * Knowledge of AI evaluation metrics, prompt monitoring, hallucination detection, and Retrieval-Augmented Generation (RAG) observability. * Experience with MLOps platforms such as MLflow, Kubeflow, SageMaker, Vertex AI, or Azure Machine Learning. * Familiarity with vector databases and model serving platforms. * Experience monitoring GPU-enabled infrastructure and AI inference systems. * Knowledge of Responsible AI, fairness monitoring, and AI governance. Technical Skills * Python * Bash * SQL * Linux * Docker * Kubernetes * AWS / Azure / Google Cloud Platform * Prometheus * Grafana * OpenTelemetry * ELK Stack * Datadog * Splunk * New Relic * MLflow * Kubeflow * SageMaker * Vertex AI * Azure Machine Learning * Evidently AI * Arize AI * Fiddler AI * WhyLabs * PostgreSQL * Redis Soft Skills * Strong analytical and troubleshooting skills * Excellent communication and collaboration * Attention to detail * Problem-solving mindset * Ownership and accountability * Continuous learning and improvement Nice to Have * Experience monitoring LLMs, AI agents, and Retrieval-Augmented Generation (RAG) systems * Knowledge of AI evaluation frameworks and model quality assessment * Experience implementing automated anomaly detection and predictive alerting * Familiarity with FinOps and AI infrastructure cost monitoring * Cloud, observability, or Kubernetes certifications Key Performance Indicators (KPIs) * AI service availability and uptime * Model inference latency and throughput * Alert accuracy and mean time to detect (MTTD) * Mean time to resolve (MTTR) production incidents * Model drift detection and response time * Dashboard coverage for AI services * Reduction in production incidents and false alerts * Infrastructure resource utilization and optimization ## Description We are seeking an AI Monitoring Engineer to design, implement, and manage monitoring solutions for AI, machine learning, and Generative AI systems. The ideal candidate will ensure the availability, reliability, performance, and quality of AI services by developing observability frameworks, monitoring model behavior, detecting anomalies, and automating alerts and incident response. This role collaborates closely with AI Engineers, ML Engineers, Data Scientists, DevOps Engineers, and Platform Engineers to maintain high-performing AI systems in production., * Design and implement monitoring frameworks for AI applications, machine learning models, and AI infrastructure. * Monitor model performance, prediction quality, latency, throughput, and availability. * Detect and analyze data drift, model drift, concept drift, and inference anomalies. * Build dashboards and alerts for AI system health using observability tools. * Develop Service Level Indicators (SLIs), Service Level Objectives (SLOs), and operational metrics for AI services. * Implement logging, distributed tracing, and telemetry for AI applications. * Configure automated alerting and incident response workflows. * Monitor GPU, CPU, memory, storage, and network utilization for AI workloads. * Analyze AI system performance trends and recommend optimization strategies. * Collaborate with AI Engineers and Data Scientists to improve model reliability and operational performance. * Support root cause analysis (RCA) for production incidents and implement preventive measures. * Maintain monitoring documentation, operational runbooks, and reporting dashboards. * Ensure compliance with monitoring, security, governance, and operational standards.