Senior AI Observability engineer

We are seeking a hands-on Senior AIOps Reliability Engineer to build the AI-native operations layer for our hybrid enterprise estate. You will design and ship LLM-based agents, retrieval-augmented knowledge pipelines, and machine-learning anomaly detection that find, triage, and remediate incidents across public cloud, private cloud, and on-premises data centers, all resting on strong SRE and network engineering fundamentals.

Our environment spans Azure, AWS, and Google Cloud alongside on-premises data centers, colocation sites, and manufacturing and HPC facilities, so cloud-agnostic design and hybrid network fluency matter more than depth in any single provider. This is a deep individual-contributor role: you will write the code, instrument the telemetry, tune the models, and own the reliability of both the infrastructure and the AI systemsoperatingon it. The impact you’ll make

Join Lam as an IT Engineer, where you’ll be at the forefront of designing, analyzing, and implementing applications and systems that form the foundation of our infrastructure. As a crucial member of our IT team, you’ll contribute your technical assistance and guidance to projects for various systems and infrastructures. Acting as a technical liaison, you’ll address complex business problems with automated systems solutions. Your expertise will be instrumental in driving Lam’s commitment to innovation and efficiency. What you’ll do

AI and Agentic Operations

• Build agentic AI workflows using LLM agents, tool and function calling, and orchestration frameworks such as LangGraph, Semantic Kernel, AutoGen, or the Model Context Protocol, applied to autonomous fault detection, triage, and remediation.
• Develop the AIOps intelligence layer: time-series anomaly detection, dynamic baselining, alert deduplication and correlation, event clustering, and predictive failure and capacity forecasting across infrastructure, network, and application telemetry from both cloud and on-premises sources.
• Engineer the retrieval knowledge fabric by chunking, embedding, and indexing runbooks, post-mortems, architecture documents, CMDB and ServiceNow records into a vector store, then tuning retrieval quality against measurable evaluations.
• Ship AI-assisted incident response: automated summarization, root-cause hypothesis generation, blast-radius analysis, and telemetry-grounded draft post-mortems wired into the paging and ITSM toolchain.
• Automate remediation safely through event-driven pipelines and configuration-management runbooks invoked by agents, with human-in-the-loop approval gates, scoped least-privilege boundaries, rollback paths, and complete audit trails for every autonomous action.
• Own AI safety and governance in production: guardrails, prompt-injection defense, hallucination and drift monitoring, PII redaction, and evaluation harnesses that gate every model or prompt change.
• Run LLMOps and MLOps, covering prompt and model versioning, offline and online evaluation, shadow and A/B testing, inference logging, token cost and latency observability, and CI/CD for every AI component.
• Instrument AI systems as first-class services with OpenTelemetry GenAI tracing, model SLOs, and quality, cost, and latency dashboards for every agent in production.

• BS, MS, or PhD in Computer Science, Engineering, or a related field, or equivalent practical experience.
• Eight or more years in SRE, DevOps, infrastructure, observability, network engineering, or platform engineering, with a track record of shipping production systems yourself.
• Production experience with LLM applications: prompt engineering, retrieval-augmented generation, embeddings and vector databases, function and tool calling, and agent orchestration.
• Practical use of machine learning for anomaly detection, forecasting, event correlation, and alert noise reduction on real operational telemetry.
• Working knowledge of at least one major AI platform and the ability to remain portable across them, including private-network deployment, quota, and cost management.
• Hands-on production experience with at least one major public cloud and the ability to design portable, cloud-agnostic patterns across the others, covering identity and access boundaries, compute, storage, managed databases, serverless functions, and event services.
• Solid on-premises infrastructure background across virtualization, storage, data-center operations, and private cloud platforms.
• Strong networking fundamentals spanning both worlds: TCP/IP, BGP and OSPF, VLANs and overlays such as VXLAN and EVPN, MPLS and SD-WAN, firewalls, load balancers, DNS, and cloud virtual network, VPC, and transit routing constructs. You can read a flow log or a routing table and reason about a failure.

CA San Francisco Bay Area Salary Range for this position: $92,000.00 -$211,000.00.

The above salary range for this position is relevant to applicants that reside or work onsite in the California, San Francisco Bay Area only. Salary offers will depend on factors that include the location you work from, your level, education, training, specific skills, years of experience and comparison to other employees already in this role. Actual salary may vary from salary offered due to numerous factors including but not limited to unpaid time off, unpaid leave, company mandated shutdown, and other relevant factors.

We believe it is important for every person to feel valued, included, and empowered to achieve their full potential. By bringing unique individuals and viewpoints together, we achieve extraordinary results.