> Markdown version of [/jobs/ext/4771591-observability-scale-engineer](https://www.wearedevelopers.com/jobs/ext/4771591-observability-scale-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Observability & Scale Engineer - **Company:** Kyndryl, Inc. - **Location:** San Francisco, CA, United States (Remote available) - **Experience:** Experienced - **Salary:** $115,080 - $218,520 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Cloud Computing, InfiniBand, Reliability Engineering, Prometheus, AI Infrastructure, Datadog, Graphics Processing Unit (GPU), Grafana, Kubernetes, Slurm, Splunk, Dynatrace - **Published:** November 18, 2026 - **Apply:** https://dejobs.org/x/x/B802145325B64AE38796864E9C07C47E/job/ ## About the Role · 4+ years in observability, performance engineering, capacity management, SRE, or infrastructure analytics. · Hands-on Prometheus, Grafana, time-series data, APIs, dashboarding, and alert engineering. · Ability to translate telemetry into operational, performance, and investment recommendations. Preferred Skills · NVIDIA DCGM, NVIDIA Run:ai, NCCL performance, UFM, and Spectrum-X or InfiniBand telemetry. · NVIDIA GPU utilization and workload-efficiency analysis, topology-aware monitoring, and AI capacity modeling. · Cloud observability, HPC performance engineering, network telemetry, storage analytics, or enterprise capacity management. · OpenTelemetry, Elastic, Splunk, Datadog, Dynatrace, VictoriaMetrics, Thanos, or comparable platforms. · Slurm accounting, Kubernetes metrics, FinOps, or infrastructure demand forecasting. · SRE, observability platform, or cloud-native credentials. ## Description Accountable for operational visibility and capacity planning across the full service stack. Gives leadership early warning of reliability and capacity risk and the evidence needed to optimize costly AI infrastructure and plan expansion., · Design a common telemetry model spanning power, cooling, compute, GPU, NVLink, network, storage, Kubernetes, scheduler, workload, and service layers. · Implement and maintain DCGM or equivalent GPU telemetry, Prometheus, Grafana, vendor management telemetry, log pipelines, and enterprise observability integration. · Monitor health, ECC conditions, throttling, thermals, link state, retransmissions, congestion, queue depth, storage latency, workload demand, and service SLOs. · Build topology-aware dashboards, alerts, dependency views, baseline reports, and operational or executive scorecards. · Baseline idle, steady-state, peak, degraded, and recovery conditions and correlate GPU utilization with CPU, memory, network, storage, queue, and application demand. · Define evidence-based thresholds and reduce false positives through tuning, suppression, routing, and contextual enrichment. · Forecast GPUs, fabric ports, storage bandwidth, rack power, cooling, scheduler demand, and expansion triggers. · Identify stranded capacity, bottlenecks, noisy-neighbor conditions, low utilization, and workload-placement inefficiency and recommend corrective action..