> Markdown version of [/jobs/ext/2714738-ai-enabled-observability-and-developer](https://www.wearedevelopers.com/jobs/ext/2714738-ai-enabled-observability-and-developer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI-Enabled Observability and Developer - **Company:** Palo Alto Networks - **Location:** United States - **Experience:** Expert - **Salary:** $147,000.0 - $237,500.0 - **Contract:** Permanent contract - **Skills:** Query Performance, Java (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Backup Devices, Configuration Management, Code Generation, Profiling, Databases, Continuous Integration, Software Debugging, Linux, Disaster Recovery, Distributed Systems, Domain Name System (DNS), Failover, Intrusion Detection Systems, Python (Programming Language), Node.Js, Octopus Deploy, Open Source Technology, Performance Tuning, Reliability Engineering, Prometheus, Runbook, Selenium, Service Discovery, Software Engineering, Systems Integration, Transmission Control Protocol (TCP), Load Balancing, Autoscaling, Istio, Delivery Pipeline, Large Language Models, Grafana, Multi-Agent Systems, Prompt Engineering, Mttr, Kubernetes Helm Charts, HybridCloud, Event Driven Architecture, Build Management, Git Flow, Kubernetes, Infrastructure Automation Frameworks, Deployment Automation, Playwright, Production Code, Terraform, Code Restructuring, Dynatrace - **Published:** September 4, 2026 - **Apply:** https://startup.jobs/principal-platform-engineer-observability-cipe-paloaltonetworks-com-8587955 ## About the Role Your Experience * 7+ years of software engineering, platform engineering, infrastructure engineering, or SRE experience, with significant experience building production-grade distributed systems. * Deep hands-on experience with observability systems, including metrics, logs, traces, profiling, dashboards, synthetics, alerting, and incident workflows. * Strong expertise with OpenTelemetry, including SDKs, Collector pipelines, exporters, processors, receivers, semantic conventions, and instrumentation patterns. * Strong experience with Prometheus-compatible metrics, Alertmanager, scraping, cardinality management, federation, and remote write patterns. * Hands-on experience with distributed tracing systems such as Jaeger or similar technologies. * Experience with continuous profiling technologies. * Strong experience with synthetic monitoring and proactive availability testing, including API checks, browser-based checks, blackbox monitoring, dependency checks, and integration with alerting and SLO workflows. * Strong Kubernetes experience, including workload monitoring, service discovery, operators/controllers, Helm, resource management, cluster observability, and multi-tenant platform patterns. * Strong Python engineering skills, including building internal tools, automation, integrations, services, and instrumentation libraries. * Hands-on experience building real solutions, tools, and developer workflows using modern AI coding agents such as Claude, Codex, or equivalent - including prompt design, skill/tool/MCP authoring, agent orchestration, and integrating LLMs into production engineering systems. * Practical understanding of how to design AI-friendly platforms: structured APIs, machine-readable runbooks, telemetry schemas, and skills/tools that allow both humans and AI agents to operate observability effectively. * Experience designing and operating high-scale, highly available infrastructure systems. * Strong understanding of SLOs, SLIs, error budgets, incident response, on-call practices, production readiness, and reliability engineering principles. * Experience writing clear technical design documents, RFCs, standards, operational runbooks, and architecture recommendations. * Ability to influence teams through technical depth, collaboration, mentorship, and pragmatic decision-making. Technical Skills * Observability: OpenTelemetry, Prometheus, Chronosphere, PromQL, Alertmanager, Grafana, Jaeger, OpenTelemetry Collector. * Telemetry: Metrics, logs, traces, spans, profiles, exemplars, service maps, SLOs, SLIs, error budgets, correlation IDs, semantic conventions. * Synthetics: Grafana k6, Prometheus Blackbox Exporter, Playwright, Selenium, API monitoring, browser checks, HTTP checks, gRPC checks, DNS/TCP/TLS checks, synthetic user journeys. * Kubernetes: Helm, operators, controllers, CRDs, DaemonSets, sidecars, service discovery, ingress, autoscaling, resource limits, multi-cluster observability. * Programming: Python required; Go, Java, Rust, or Node.js preferred. * AI Engineering: Claude, Codex, and equivalent coding agents; skill/tool/MCP authoring; prompt engineering; agent orchestration; LLM integration patterns; grounding, evaluation, and guardrails for AI-driven workflows. * Infrastructure: Linux, containers, networking, distributed systems, cloud platforms, service mesh, load balancers, APIs, queues, databases. * Automation: CI/CD, GitOps, Terraform, Argo CD, Flux, deployment pipelines, release validation, configuration management. * Reliability: Incident response, alert tuning, runbooks, error budgets, capacity planning, performance optimization, disaster recovery, production readiness. ## Description We are looking for a Principal Software Engineer to architect, build, and evolve our observability platform across infrastructure, applications, and developer workflows. This role is ideal for a hands-on technical leader with deep experience in open source observability technologies and Chronosphere, who is equally fluent in building AI-enabled systems and developer experiences using modern AI coding tools such as Claude and Codex. You will serve as a technical architect for the observability stack, working across engineering, platform, SRE, and product teams to define standards for metrics, logs, traces, profiling, synthetics, alerting, dashboards, and incident response. You will also lead the integration of AI agents, copilots, and skill-based automation into observability workflows - making telemetry, debugging, and reliability operations equally consumable by humans and AI agents. You should be comfortable operating at both strategic and implementation levels: designing architecture, writing production-grade code, reviewing systems, mentoring engineers, and driving adoption across teams. Your Impact Observability Architecture * Design and lead the evolution of a modern observability platform using OpenTelemetry, Prometheus, Jaeger, Alertmanager, and related CNCF ecosystem tools. * Define architecture standards for telemetry collection, processing, storage, querying, visualization, alerting, retention, and governance. * Build scalable systems for metrics, distributed tracing, continuous profiling, log aggregation, synthetic monitoring, service health monitoring, and reliability analytics. * Establish best practices for instrumentation across services, infrastructure, Kubernetes workloads, CI/CD systems, and developer platforms. * Evaluate trade-offs around data cardinality, sampling, storage cost, retention, query performance, multi-tenancy, reliability, and operational complexity. * Make pragmatic recommendations on open source, self-managed, managed-service, and hybrid observability approaches. * Create paved-road observability patterns that help engineering teams instrument, monitor, debug, and operate services with minimal friction. OpenTelemetry and Instrumentation * Lead adoption and standardization of OpenTelemetry across applications, services, infrastructure, and platform components. * Design and implement telemetry pipelines using OpenTelemetry Collector, exporters, processors, receivers, connectors, and custom extensions where needed. * Define conventions for traces, metrics, logs, spans, attributes, resources, service names, correlation IDs, and semantic conventions. * Build libraries, SDK wrappers, golden paths, and internal tooling to simplify observability instrumentation for engineering teams. Metrics, Monitoring, and Alerting * Architect metrics systems using Prometheus-compatible formats, PromQL, remote write, federation, scraping strategies, service discovery, recording rules, and long-term storage backends. * Design alerting frameworks that reduce noise, improve signal quality, and align with SLOs, SLIs, error budgets, and incident response practices. * Create reusable alerting patterns for Kubernetes, infrastructure, applications, APIs, databases, queues, event-driven systems, and distributed services. * Define standards for dashboarding, runbooks, escalation policies, alert ownership, and production readiness. * Partner with SRE and engineering teams to mature monitoring practices and improve service reliability. Kubernetes and Platform Engineering * Build observability capabilities for Kubernetes environments, including cluster monitoring, workload telemetry, service mesh visibility, ingress and egress monitoring, and node-level insights. * Develop and maintain Helm charts, Kubernetes manifests, operators, sidecars, agents, DaemonSets, and deployment automation for observability components. * Work with platform teams to ensure observability systems are reliable, secure, multi-tenant, highly available, and easy to operate. * Define standards for resource usage, scaling, upgrades, failover, backup, disaster recovery, access control, and tenant isolation for observability infrastructure. * Support observability across multi-cluster, multi-region, and hybrid cloud environments where applicable. AI-Enabled Observability and Developer Experience * Design and build AI-enabled observability workflows that allow both humans and AI agents to investigate incidents, query telemetry, summarize signals, and propose remediations. * Define and publish reusable AI skills, agents, and tools (e.g., Claude skills, Codex tools, MCP servers, structured prompts) that encode observability best practices and make platform capabilities consumable by engineering teams and autonomous agents. * Build paved-road AI integrations for triage, alert summarization, root-cause analysis, log/trace exploration, runbook generation, dashboard authoring, and post-incident review. * Establish standards for grounding AI agents in authoritative telemetry, runbooks, and service catalogs, with strong guardrails around accuracy, safety, cost, and auditability. * Use AI coding tools (Claude, Codex, and equivalents) as a first-class part of the engineering workflow - for code generation, refactoring, instrumentation rollouts, migrations, and platform automation - and define patterns the broader team can adopt. * Partner with platform, SRE, and product teams to evolve observability from human-only dashboards toward agent-assisted, self-serve reliability operations., * The organization has a clear, scalable observability architecture with strong standards for telemetry generation, collection, storage, querying, retention, and consumption. * Engineering teams can easily instrument services and get useful metrics, traces, profiles, logs, dashboards, synthetic checks, and alerts without deep observability expertise. * Alerting becomes more actionable, less noisy, and better aligned with service health, SLOs, and customer impact. * Synthetic monitoring proactively detects failures in critical user journeys, APIs, infrastructure endpoints, and third-party dependencies before customers are significantly impacted. * The observability platform is reliable, cost-efficient, secure, multi-tenant, and easy to operate across Kubernetes environments. * Continuous profiling and tracing become part of normal performance, debugging, and reliability workflows. * AI agents and skills are first-class consumers of the observability platform - accelerating triage, investigation, and remediation for both humans and autonomous workflows, with measurable improvements in MTTR and engineer productivity. * The Principal Engineer is recognized as the technical leader who can connect architecture, implementation, operational excellence, developer experience, AI-enabled workflows, and business reliability outcomes across the observability stack. ## Related Videos - [Unlocking the AI Black Box: Building Trust in the Era of Agentic Production](https://www.wearedevelopers.com/videos/100086-unlocking-the-ai-black-box-building-trust-in-the-era-of-agentic-production) - [Docker network without Docker](https://www.wearedevelopers.com/videos/1418-docker-network-without-docker) - [What Developers Get Wrong About Application Quality](https://www.wearedevelopers.com/videos/233-what-developers-get-wrong-about-application-quality) - [Rate-limiting using eBPF and Istio: How to protect your SaaS customers from themselves](https://www.wearedevelopers.com/videos/100220-rate-limiting-using-ebpf-and-istio-how-to-protect-your-saas-customers-from-themselves) - [Docker exec without Docker](https://www.wearedevelopers.com/videos/1094-docker-exec-without-docker) - [Get started with securing your cloud-native Java microservices applications](https://www.wearedevelopers.com/videos/123-get-started-with-securing-your-cloud-native-java-microservices-applications) ## Related Articles - [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)