AI Infrastructure Engineer

We are looking for an AI Infrastructure Engineer to own the entire software stack of our GPU clusters - from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups - our foundation model training teams and our model serving/product teams - ensuring both run on correctly configured, well-managed, high-performance infrastructure., * OS Bring-Up & Node Lifecycle Engineering

• Golden Images & Automated Bring-Up: Own the node software definition - versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks - and the automated pipeline that takes a node from base OS to production-ready.
• Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.
• Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver CUDA/ROCm framework compatibility matrix across the fleet.
• Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.
• Configuration Management & Automation
• Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.
• Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.
• Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.
• Orchestration & Scheduling (Kubernetes & Slurm)
• Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads - NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.
• Training Schedulers: Operate Slurm (or Run:AI) for multi-node training - partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).
• Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.
• GPU Driver & ML Stack Engineering
• Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack - NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) - including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).
• Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.
• Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.
• Advanced Debugging & Observability
• Escalation Point: Own the hard problems - NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.
• Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting.

• 5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience.
• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
• Deep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning.
• Hands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging.
• Production Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) - or the reverse (deep Slurm, working K8s).
• Strong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows.
• Provisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds.
• Client-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization.
• Container fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent.
• Proficiency in Python and Bash for automation and tooling.
• Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior.
• Nice-to-Haves:
• Experience directly supporting foundation model training teams - multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage - ideally in a startup or research-heavy environment.
• Experience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets.
• GPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF.

Pulled from the full job description

• 401(k)
• Health insurance
• Vision insurance
• Dental insurance, * Medical, dental, and vision insurance
• 401k plan
• Daily lunch, snacks, and beverages
• Flexible time off
• Competitive salary and equity