> Markdown version of [/jobs/ext/2282378-principal-software-engineer-at-scale-reliability-and-fleet-intelligence-csp-engagements](https://www.wearedevelopers.com/jobs/ext/2282378-principal-software-engineer-at-scale-reliability-and-fleet-intelligence-csp-engagements). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Software Engineer, At-Scale Reliability and Fleet Intelligence - CSP Engagements - **Company:** NVIDIA Corporation - **Location:** Santa Clara, CA, United States - **Salary:** $272,000.0 - **Contract:** Permanent contract - **Skills:** Application Integration Architecture, Databases, Firmware, Pattern Recognition, Reliability Engineering, Software Engineering, Data Streaming, System Software, Network Switches, Information Technology - **Published:** August 28, 2026 - **Apply:** https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Principal-Software-Engineer--At-Scale-Reliability-and-Fleet-Intelligence---CSP-Engagements_JR2020320 ## About the Role * 15+ years of experience in systems software at datacenter scale, or reliability engineering with focus on at-scale challenges. * BS or MS in Computer Science, Electrical Engineering, Statistics, or related field (or equivalent experience) * Deep expertise in multi-NUMA, rack-scale system software and firmware. Statistical failure analysis methods: MTBF/MTBI calculation, Pareto analysis, root cause classification * Experience with fleet-level telemetry and observability systems: time-series databases, anomaly detection, health scoring, event correlation * Understanding of hardware failure modes in large-scale GPU/accelerator deployments - ability to classify and prioritize across compute, interconnect, memory, power, and thermal domains * Experience defining or operating burn-in, stress testing, or certification frameworks for complex hardware systems. Familiarity with predictive maintenance or anomaly detection approaches applied to fleet health data * Customer obsession - genuine passion for understanding fleet reliability challenges at scale and translating them into actionable engineering priorities * Strong communication - ability to present statistical reliability findings to both deep technical audiences and executive leadership. Demonstrated success driving cross-functional improvements across hardware, firmware, and software teams without direct authority, * Experience in fleet reliability at a hyperscaler (hardware health, fleet reliability at leading CSP/Hyperscaler) * Familiarity with NVIDIA GPU error taxonomy (Xid errors, NVLink error counters, thermal events, CPER records) * Experience building health scoring or predictive failure models for accelerator or HPC infrastructure * Background in defining MTBI/MTBF measurement standards or certification programs for complex multi-component systems * Understanding of how reliability data flows from device firmware through telemetry pipelines to fleet-level dashboards and automated remediation ## Description We're looking for a Principal Software Engineer to join our CSP Engagements team as the technical focal point for fleet-scale reliability, working directly with engineering teams of key CSP / hyperscale customers to ensure NVIDIA platforms achieve target MTBI (Mean Time Between Interruptions) in production. In this role, you will augment NVIDIA's internal software/firmware and quality teams with a dedicated CSP-facing focus. You will drive work streams with CSP engineering teams to build shared understanding of reliability software/firmware architecture, methodology, incorporate their fleet telemetry and failure data into NVIDIA's improvement priorities, and validate that reliability improvements measured in the lab translate to real customer environments. Your cross-CSP visibility enables you to distinguish systemic architectural gaps from environmental or configuration-specific issues that no single customer engagement could identify alone. What you'll be doing: * Drive reliability work streams with CSP engineering teams - ensuring shared understanding of MTBI measurement methodology, failure classification, and health monitoring architecture * Gather and synthesize CSP fleet reliability data - identify failure patterns that appear across multiple customers and champion improvements back into NVIDIA's firmware, driver, and hardware teams * Define consistent MTBI measurement methodology that works across different CSP monitoring environments and operational practices * Conduct fleet-scale failure pattern analysis using statistical methods (Pareto, survival analysis, Weibull) to classify failures as systemic, environmental, or configuration-specific * Drive fleet health monitoring integration architecture - ensure NVIDIA's health agents, telemetry, and reporting align with CSP operational workflows and automation * Define burn-in reliability test environment and cluster certification criteria in collaboration with quality teams, validating with customers that criteria are meaningful * Collaborate with CSPs to ensure reliability-related integration work (health monitoring deployment, telemetry pipeline, alerting configuration) is complete ahead of at-scale launch * Develop predictive failure models using fleet telemetry and validate their effectiveness in customer environments ## Related Videos - [Coffee with Developers - Stephen Jones - NVIDIA](https://www.wearedevelopers.com/videos/1303-coffee-with-developers-stephen-jones-nvidia) - [Python-Based Data Streaming Pipelines Within Minutes](https://www.wearedevelopers.com/videos/1233-python-based-data-streaming-pipelines-within-minutes) - [Playing Pong on a shoulder press machine](https://www.wearedevelopers.com/videos/100140-playing-pong-on-a-shoulder-press-machine) - [Kubernetes and Microservices with Multi-Model Databases](https://www.wearedevelopers.com/videos/382-kubernetes-and-microservices-with-multi-model-databases) - [Software stack under and over the hood of the fastest accelerating car in the world](https://www.wearedevelopers.com/videos/221-software-stack-under-and-over-the-hood-of-the-fastest-accelerating-car-in-the-world) - [Accelerating Python on GPUs](https://www.wearedevelopers.com/videos/859-accelerating-python-on-gpus) ## Related Articles - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [How software is steering vehicle technology](https://www.wearedevelopers.com/magazine/515-how-software-is-steering-vehicle-technology) - [Trustworthy AI Starts at Deployment: 5 Checks Before You Ship](https://www.wearedevelopers.com/magazine/753-trustworthy-ai-starts-at-deployment-5-checks-before-you-ship) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)