> Markdown version of [/jobs/ext/4481908-principal-engineer-rack-scale-gpu-architecture](https://www.wearedevelopers.com/jobs/ext/4481908-principal-engineer-rack-scale-gpu-architecture). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Principal Engineer, Rack-Scale GPU Architecture

- **Company:** DigitalOcean, LLC
- **Location:** Seattle, WA, United States
- **Salary:** $249,600.0 - $312,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Network Congestion, Software Debugging, Firmware, InfiniBand, Python (Programming Language), Node.Js, Open Source Technology, PCI Express, Remote Direct Memory Access, Digitalocean, System Programming, Non-Uniform Memory Access, Kubernetes, Hardware Infrastructure
- **Published:** October 7, 2026
- **Apply:** https://www.dice.com/job-detail/5b418387-de0a-4c9a-a29c-527d9ac82412

## About the Role

* 12+ years in large-scale compute, HPC, or GPU infrastructure engineering, including hands-on responsibility for systems in production
 * Deep understanding of GPU interconnect and system topology-NVLink and NVSwitch, NVLink domains, PCIe, NUMA, and how each shapes what a workload can be placed where
 * Strong RDMA networking background: RoCEv2 or InfiniBand at scale, GPUDirect RDMA, congestion control and lossless fabric tuning, and rail-optimized cluster topologies
 * Practical NCCL expertise-not just running it, but debugging it: algorithm and protocol selection, topology detection, and diagnosing collectives that are slow for non-obvious reasons
 * Substantial Kubernetes depth for accelerated workloads: device plugins and DRA, scheduling extensions, operators, and a clear-eyed view of where Kubernetes' model breaks down against rack-scale hardware
 * Strong systems programming and automation skills in Go and Python, and comfort at the driver, firmware, and BMC layer when the problem lives there
 * Experience operating fleets where hardware failure is routine, including the diagnostics, automation, and blast-radius thinking that makes that survivable
 * Excellent written and verbal communication, and a track record of leading architecture across organizational boundaries-hardware, networking, platform, and product

Bonus:

 * Direct experience bringing up GB200 or GB300 NVL72 systems, or comparable rack-scale platforms, in a production environment
 * Experience with multi-node inference for very large models-expert or pipeline parallelism spanning NVLink domains, and the placement constraints that follow
 * Familiarity with liquid-cooled and high-density rack deployments, and the operational realities of servicing them
 * Contributions to relevant open source: Kubernetes SIG-Node or scheduling, NVIDIA GPU or Network Operator, DRA drivers, Kueue, LeaderWorkerSet, or NCCL
 * Experience running heterogeneous fleets spanning multiple accelerator vendors and generations under a single orchestration model, * We innovate with purpose. You'll be a part of a cutting-edge technology company with an upward trajectory, who are proud to simplify cloud and AI so builders can spend more time creating software that changes the world. As a member of the team, you will be a Shark who thinks big, bold, and scrappy, like an owner with a bias for action and a powerful sense of responsibility for customers, products, employees, and decisions.

## Description

* Defining how NVLink domains are partitioned, allocated, and isolated across tenants and workloads, including multi-node NVLink and IMEX domain provisioning, fabric manager configuration, and the blast radius each choice implies
 * Architecting the scale-out RDMA fabric that connects racks-RoCEv2 or InfiniBand, rail-optimized topologies, congestion control, GPUDirect RDMA paths-and tuning NCCL to actually use it well
 * Making Kubernetes topology-aware for these systems: DRA for GPU and NVLink-domain allocation, multi-node inference primitives such as LeaderWorkerSet, gang scheduling, Topology Manager and NUMA alignment, and the GPU and Network Operators that underpin all of it
 * Driving multi-node disaggregated serving on this hardware in partnership with the inference platform team-how prefill and decode pools map onto NVLink domains and rail boundaries, and where the KV cache moves between them
 * Owning failure semantics and serviceability at rack scale: health checking and DCGM-based diagnostics, drain and repair workflows, degraded-domain scheduling, and firmware and driver lifecycle across a fleet of racks
 * Establishing the validation and burn-in methodology that qualifies a rack for production-collective bandwidth and latency characterization, NCCL and end-to-end inference benchmarks, and the acceptance gates a rack must pass
 * Partnering with our data center engineering team on power, liquid cooling, and rack density constraints, and translating those into what is actually deployable and at what cost
 * Working directly with NVIDIA and ODM partners on pre-silicon enablement, early hardware bringup, and roadmap feedback
 * Setting technical direction, mentoring senior and staff engineers across teams, and representing DigitalOcean upstream, at conferences, and in customer architecture reviews