> Markdown version of [/jobs/ext/4013251-software-engineer-performance-instrumentation](https://www.wearedevelopers.com/jobs/ext/4013251-software-engineer-performance-instrumentation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer, Performance Instrumentation

- **Company:** NVIDIA Corporation
- **Location:** Santa Clara, CA, United States
- **Experience:** Expert
- **Salary:** $168,000.0 - $270,250.0
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, C++ (Programming Language), Code Review, Nvidia CUDA, Computer Engineering, Programming Tools, Network Interface Controllers, Python (Programming Language), Machine Learning, Tensorflow, Software Engineering, Pytorch, Information Technology
- **Published:** October 29, 2026
- **Apply:** https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/US-CA-Santa-Clara/Software-Engineer--Performance-Instrumentation_JR2028980-1

## About the Role

* Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
 * 8+ years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
 * Proficiency in CUDA, and working knowledge of machine learning frameworks like PyTorch or JAX, as well as annotation or tracing APIs such as NVTX.
 * Demonstrated experience in maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
 * Proven ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
 * Proven history of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion through to delivery.
 * Strong debugging skills across component boundaries and the ability to validate changes through code review, tests, and independent measurements, including using automated development tools.
 * Excellent communication skills, especially in conveying evidence, uncertainty, and next steps with engineers and data users.
 * Proficiency in Python and C++.

Ways to stand out from the crowd:

 * Experience establishing telemetry conventions or schemas adopted by other teams.
 * Background in storage, checkpointing, or data-loading performance.
 * Expertise in crafting low-overhead event collection.
 * Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.

## Description

NVIDIA is looking for a Software Engineer, Performance Instrumentation to join our outstanding team in Santa Clara, CA. This is an outstanding opportunity to be part of a world-class organization, driving innovation at the intersection of AI and accelerated computing. As a part of our team, you will work on groundbreaking projects and have a direct impact on the future of technology.

What you'll be doing:

 * Define and maintain the performance data requirements from various sources including profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
 * Identify and prove visibility gaps with concrete evidence, demonstrating which diagnoses are not possible with current data. Collaborate with relevant teams to address these gaps or build solutions where necessary.
 * Inform the build of low-overhead instrumentation and annotation APIs from the perspective of the consumer, ensuring the necessary information is conveyed efficiently and effectively.
 * Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting mechanisms, so that any gaps or errors are detected before impacting users or automated analysis.
 * Measure and manage the overhead and observer effects of instrumentation on real workloads, ensuring they remain within acceptable limits.
 * Collaborate with multiple partner teams simultaneously, managing commitments and ensuring continuous progress.