Senior System Software Engineer, DGX Cloud AI Infrastructure

We’re looking for a Senior Software Engineer for the NVIDIA Recovery and Resilience team. You’ll own our AI infrastructure services, tools and open-source projects to support resilient and efficient training from design through production. You’ll also mentor engineers and decide how the team runs what it builds.

What You Will Be Doing:

• Drive the design, build, launch and operation of our services, tools and open-source projects.
• Plan and assign the team’s work, mentor junior engineers and review designs.
• Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA’s own training workloads and customer workloads. This includes supercomputers at major cloud providers and on-premises clusters.
• Build APIs that connect to NVIDIA’s resiliency systems: telemetry, break/fix automation, and checkpoint and execution systems.
• Build a job submission layer that hides infrastructure failures from users and restarts long-running training jobs automatically.
• Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures and recommending fixes.
• Drive our open-source work. Release components so customers and the community can use them, and work with them to improve those components.
• Set up how the team runs its services: support, monitoring, alerts, CI/CD, integration test gates and on-call plans.

• BS, MS or PhD in Computer Science or a related field (or equivalent experience), plus 12+ years of experience.
• Proven experience who has taken large systems from design through production.
• Strong experience with Kubernetes and/or Slurm at scale.
• Hands-on experience with PyTorch, TensorFlow, JAX or Ray.
• Strong understanding of NVIDIA GPUs and networking, including how they fail.
• Experience running production services, including CI/CD, monitoring, alerting and on-call.

Ways to Stand Out from the Crowd:

• Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.

• Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
• Background in building external services or maintaining open-source projects in ML infrastructure.

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are: 224,000 USD - 356,500 USD for Level 5; and 272,000 USD - 431,250 USD for Level 6.

You will also be eligible for equity and benefits.