Cluster Engineer

We are seeking a highly experienced AI Infrastructure Engineer to architect, deploy, optimize, and operate large-scale GPU clusters supporting state-of-the-art AI training and inference workloads. This is a deeply technical role focused on maximizing cluster efficiency, scalability, and performance across the entire AI stack-from GPU hardware and high-speed networking to distributed training frameworks and inference optimization. The ideal candidate has built GPU clusters from the ground up, tuned distributed training environments, optimized large-scale inference deployments, and understands how every layer of the infrastructure contributes to application performance.

Responsibilities

• Design, deploy, and optimize multi-node GPU clusters for AI training and inference workloads.
• Tune distributed training environments to maximize GPU utilization, throughput, and scaling efficiency.
• Optimize inference clusters for maximum token generation throughput, low latency, and high GPU utilization.
• Build and support production AI infrastructure running hundreds to thousands of GPUs.
• Analyze and eliminate performance bottlenecks across compute, networking, storage, and software layers.
• Perform NCCL benchmarking, analysis, and tuning to achieve optimal collective communication performance.
• Design and optimize GPU networking using InfiniBand or RoCE v2, including RDMA, congestion management, topology awareness, and QoS.
• Configure and tune distributed AI software stacks including:
• PyTorch
• NCCL
• CUDA
• UCX
• MPI
• Slurm
• Pyxis/Enroot
• Optimize GPU scheduling and resource allocation for both training and inference environments.
• Develop repeatable benchmarking and validation processes for new hardware, firmware, drivers, and software releases.
• Identify performance regressions and troubleshoot distributed training issues at scale.
• Optimize storage architectures for AI workloads, including checkpointing, dataset streaming, and high-performance parallel I/O.
• Work closely with ML engineers to improve training scalability and inference efficiency.
• Create automation to deploy, validate, benchmark, and monitor GPU clusters.
• Evaluate emerging AI infrastructure technologies and recommend improvements to platform architecture.

• 7+ years designing or operating large-scale Linux infrastructure.
• 5+ years supporting production GPU clusters for AI or HPC workloads.
• Demonstrated experience building multi-node GPU training environments from the ground up.
• Deep expertise with distributed PyTorch training.
• Extensive experience troubleshooting and optimizing NCCL communications.
• Strong understanding of distributed AI communication patterns, including:
• AllReduce
• ReduceScatter
• AllGather, * Strong understanding of GPU memory management, including:
• KV Cache
• Activation checkpointing
• Tensor Parallelism
• Pipeline Parallelism
• Data Parallelism
• Experience optimizing LLM inference throughput, including:
• Tokens/sec optimization
• Batch sizing
• Continuous batching
• KV cache tuning
• Memory bandwidth optimization
• Experience tuning CUDA, NCCL, UCX, and MPI for maximum distributed performance.
• Expert-level Linux systems administration skills.
• Experience with Slurm workload manager.
• Experience using Pyxis and Enroot for containerized GPU workloads.
• Strong scripting skills using Python and Bash.

Technical Expertise AI Frameworks

• PyTorch
• CUDA
• NCCL
• Triton (preferred)
• TensorRT-LLM (preferred)

Cluster Scheduling

• Slurm
• Pyxis
• Enroot

GPU Networking Strong understanding of:

• InfiniBand
• RoCE v2
• RDMA
• GPUDirect RDMA
• GPUDirect Storage
• UCX
• MPI
• Network topology optimization
• Congestion control
• QoS
• ECN/PFC
• High-speed Ethernet (200/400/800 GbE)

Storage Experience designing or tuning storage for AI workloads, including:

• Parallel file systems
• Distributed storage
• Object storage
• NVMe
• Checkpoint optimization
• Dataset staging
• GPUDirect Storage
• Storage bandwidth optimization
• Metadata performance

Performance Engineering Experience with:

• NCCL benchmarking
• Multi-node scaling analysis
• GPU utilization optimization
• Communication/computation overlap
• NUMA optimization
• CPU affinity
• PCIe topology
• GPU topology (NVLink/NVSwitch)
• Memory bandwidth analysis
• End-to-end performance profiling

Preferred Qualifications

• Experience deploying AI workloads on Kubernetes.
• Experience with NVIDIA GPU Operator.
• Experience with Kubernetes batch scheduling (Volcano, Kueue, Run:ai, etc.).
• Experience with distributed inference platforms such as vLLM, TensorRT-LLM, or SGLang.
• Experience with NVIDIA DGX SuperPOD or similar large-scale GPU deployments.
• Familiarity with MLPerf benchmarking.
• Experience deploying monitoring solutions such as Prometheus, Grafana, and DCGM Exporter.
• Experience automating infrastructure using Ansible, Terraform, or similar tools.
• Experience working in cloud GPU environments (AWS, Azure, GCP) in addition to bare metal.