AI-Powered Incident Triage: How We Built GenAI Agents with MCPs to Automate On-Call Workflows

Modern software systems generate incident noise at a volume that overwhelms traditional on-call rotations. In this talk, I will describe how a well designed fleet of GenAI agents are connected to live operational tooling via the Model Context Protocol (MCP) to automate the triage, enrichment, and routing of production incidents at scale. The agentic pipeline classifies and deduplicates incoming alerts, autonomously queries observability platforms, runbooks, and dependency graphs to build context, and routes incidents to the right team with a proposed remediation path. The lessons learned from operating in production, including how to handle low-confidence decisions, maintain human-in-the-loop escalation paths, and measure reliability against ground-truth data, alongside results showing meaningful reductions in mean time to acknowledge (MTTA) and alert fatigue across on-call rotations.