Senior Architect - AI Infrastructure & Fabric

MONDO SOLUTIONS, LLC
Plano, TX, United States
26 days ago
Apply on www.indeed.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
6 years minimum
Compensation
$187,200.0 - $249,600.0
Working hours
Regular working hours
Job source

Tech stack

Artificial Intelligence Airflow Network Congestion Data Centers Linux Ethernet Firmware General Parallel File Systems InfiniBand Subnetting PCI Express Performance Tuning
+7 more
Remote Direct Memory Access Weka AI Infrastructure Non-Uniform Memory Access Storage Technologies Bare Metal Nvme

Job description

You own everything beneath the operating system: GPU node architecture, the compute and storage fabrics, bare-metal provisioning, high-performance storage design, and the facility constraints that decide whether an AI factory is buildable. This is the role that makes clusters scale linearly instead of falling over at sixteen nodes - and the role customers rely on when their training job stops scaling and nobody knows why.

What you’ll do

  • Design GPU cluster physical and logical topology: node configuration, rail-optimized fabric layout, oversubscription ratios, failure domains.
  • Architect and deploy InfiniBand (NDR/XDR) and high-performance Ethernet fabrics - subnet management and UFM, adaptive routing, congestion control, SHARP offload.
  • Own high-performance storage architecture for AI: parallel filesystems (WEKA, Lustre, GPFS, VAST, DDN), GPUDirect Storage, NVMe-oF and NFS-over-RDMA, and sizing for dataloader read, checkpoint burst write, and artifact serving.
  • Own bare-metal lifecycle - cluster provisioning (Bright Cluster Manager or equivalent), firmware and driver baselines, imaging, node validation and burn-in.
  • Produce BOMs and sizing for compute, network, optics, and storage; validate against customer power, cooling, floor loading, and lead times.
  • Run scaling and fabric benchmarks (NCCL bus bandwidth, IB performance tests, IOR/fio) and own cluster acceptance criteria.
  • Diagnose interconnect and scaling faults: link errors, topology binding, NUMA and PCIe affinity, GPUDirect RDMA verification, I/O stalls.
  • Set infrastructure standards for the offshore delivery pod and review their work before customer delivery.

Requirements

  • 6+ years HPC or AI infrastructure engineering; production multi-node GPU cluster experience is mandatory.
  • Deep InfiniBand and/or high-performance Ethernet: fabric design, subnet management, congestion behavior, and troubleshooting at scale.
  • NVIDIA GPU platforms - HGX or DGX-class systems, NVLink and NVSwitch topology, driver and firmware stack.
  • Parallel or scale-out filesystem design and performance tuning for I/O-intensive workloads.
  • Bare-metal cluster provisioning and deep Linux systems engineering.
  • Data center fundamentals: rack power, airflow, direct-liquid cooling concepts, cabling and optics planning., * HPC or AI infrastructure engineering: 6 years (Preferred)

Benefits & conditions

NVIDIA Enterprise Reference Architecture exposure; Spectrum-X or BlueField DPU; liquid-cooled deployment experience; WEKA/VAST/DDN certification; NVIDIA networking certification.

First 90 days

Reference cluster built and benchmarked; fabric and storage design standards published; automated node validation suite in place.

Pay: $90.00 - $120.00 per hour

Application Question(s):

  • Are you amenable working on a hybrid work set up?
  • Are you comfortable working in a contract-based role?
  • How much is your expected hourly rate?
  • What is your earliest available start date?

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.indeed.com
Prepare application

Good distractions

Loading talks and stories from around this role…