> Markdown version of [/jobs/ext/4008806-software-engineer-ml-performance](https://www.wearedevelopers.com/jobs/ext/4008806-software-engineer-ml-performance). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer, ML Performance

- **Company:** NVIDIA Corporation
- **Location:** Santa Clara, CA, United States
- **Salary:** $224,000.0
- **Contract:** Permanent contract
- **Skills:** Nvidia CUDA, Computer Engineering, Extract Transform Load (ETL), Software Debugging, Programming Tools, Distributed Computing Environment, Python (Programming Language), Software Engineering, System Software, Graphics Processing Unit (GPU), Pytorch, Large Language Models, Information Technology, SGLang, TensorRT, VLLM, Megatron
- **Published:** October 29, 2026
- **Apply:** https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Software-Engineer--ML-Performance_JR2028979

## About the Role

* Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
 * 12+ years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production.
 * Demonstrated performance work across more than one layer of the stack, with a clear account of the baseline, evidence, alternative explanations and validated outcome.
 * Experience diagnosing performance across many jobs, machines or tenants, not only tuning a single workload.
 * Enough familiarity with framework, runtime and compiler interfaces to say where a signal should be measured, even when someone else writes the instrumentation.
 * Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and the effects of the measurement itself, and know when the evidence is insufficient.
 * Strong programming and debugging skills, demonstrated by maintainable analysis code, tests or repeatable investigation procedures, including when using automated development tools.
 * Clear communication of findings, trade-offs and remaining uncertainty to the people who own the workload.
 * Proficiency with Python and CUDA.

Ways to stand out from the crowd:

 * Improved operation of both training and inference.
 * Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang or Dynamo.
 * Experience moving between application, system software, driver or hardware roles, and the full-stack view that comes with it.
 * Statistical judgment applied to noisy, multi-node fleet-scale performance data. Ideally, experience with counter-factual analysis and segmentation of workloads for diagnosing regressions.
 * Built debugging or performance-analysis tools used by other engineers, with evidence that they made investigations more effective.

## Description

* Analyze the performance of distributed training and inference workloads throughout the entire stack: model and framework, data loading and checkpointing, compiler and kernels, CUDA and NCCL, host and network. Identify the bottleneck, resolve it or suggest a solution, and provide the supporting evidence.
 * Define the performance metrics tracked across a fleet of GPUs. Break them down into factors engineers need to debug. Decide which signals the platform collects first.
 * Drive the analysis of regressions: segment workloads across machines, versions, and configurations to isolate the cause, and turn the method into analysis others can reuse.
 * Evaluate automated diagnoses: judge whether a finding is accurate, identify misleading results and missed problems, and build the labeled cases that keep the analysis honest.
 * Work closely with build partners on their real workloads, carry findings through to improvements, and feed what the product is missing back into the roadmap.