Principal Site Reliability Engineer

This is a principal-level, deeply hands-on SRE role and the most senior individual contributor responsible for the operational reliability of our client's AI systems, agents, platform services, and infrastructure in production. This role owns the SLO framework that defines reliability, the machine-readable runbook library that operationalizes incident response, the Prometheus observability configuration, and the incident response process. The Principal SRE writes SLOs, builds alerting rules, authors machine-readable runbooks, configures Prometheus, and owns the reliability engineering practices for AI systems in an enterprise production environment., Own and continuously evolve the SLO framework, defining and maintaining Service Level Objectives, Service Level Indicators, and error budget policies for all AI services Own and maintain the machine-readable runbook library, authoring and operating structured runbooks that enable automated incident response workflows Own the Prometheus observability configuration, including scrape configurations, recording rules, alerting rules, Alertmanager routing, and Grafana dashboard architecture Own and continuously improve the incident response process, defining the lifecycle, severity classification, escalation paths, and post-incident review process Design and enforce reliability engineering standards across the engineering organization Build and maintain toil elimination automation to reduce operational burden Partner with Principal Platform and AI Platform Engineers to instrument Azure infrastructure and design AI-specific reliability patterns Lead post-incident reviews, facilitating blameless retrospectives and tracking remediation commitments Mentor platform and application engineers on SRE principles, reliability design, and observability best practices

8+ years of site reliability, platform, or production operations engineering experience 3+ years of principal or senior-level ownership of SRE programs or enterprise observability platforms Deep expertise in Prometheus, Grafana, Alertmanager, and the broader Prometheus ecosystem (PromQL, recording rules, alerting rule design, etc.) Demonstrated experience designing and operating SLO frameworks Experience designing and maintaining machine-readable runbook libraries Strong experience owning incident response processes Experience designing and implementing toil elimination automation for complex distributed systems Ability to set reliability standards and influence engineering architecture decisions Excellent written and verbal communication skills for documentation, incident reports, and cross-team collaboration Desired skills: Experience defining and operating SLOs for AI systems, LLM-powered applications, or agentic workflows Experience with Azure Monitor, Azure Managed Grafana, or Container Insights Experience with PagerDuty, OpsGenie, or comparable incident management platforms Experience in financial services, cybersecurity, or other regulated enterprise environments Experience with chaos engineering practices (Chaos Mesh, Azure Chaos Studio) Familiarity with OpenTelemetry for distributed tracing and metric collection Experience with DORA metrics tracking and reliability scorecards Google SRE certification, Prometheus Certified Associate (PCA), or comparable reliability and observability certifications