AI Infrastructure Deployment Engineer

InOpTra Digital
Union City, CA, United States
3 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Working hours
Regular working hours
Job source

Tech stack

Artificial Intelligence Airflow Data Centers Linux Ethernet Network Interface Controllers Firmware InfiniBand Network Control Node.Js AI Infrastructure Scripting
+3 more
Graphics Processing Unit (GPU) System Availability Slurm

Job description

Builds what the architect designed, and proves it works. Hands-on from rack and stack through to signed-off acceptance: physical installation, firmware and out-of-band configuration, fabric bring-up, cluster software install, and the validation and burn-in that demonstrates the cluster performs to specification. This is a build-and-prove role. The validation half matters as much as the installation half, because an OEM-branded GPU build does not come with a ready-made acceptance suite - NVIDIA certification validates server models and configurations, not the cluster standing on a customer floor. The engineer runs and, where needed, assembles that acceptance testing. Responsibilities Physical build *

  • Install GPU servers and GPUs; verify seating, airflow and serviceability.
  • Identify and validate cables and transceivers across the compute, storage and management fabrics; confirm signal quality before handover.
  • Confirm power and cooling parameters against the design, and escalate where the data hall will not support it.

Bring-up and configuration *

  • Configure BMC and out-of-band management; set up TPM where required.
  • Apply and verify firmware across GPU baseboards, NICs, switches, DPUs and transceivers; interpret and act on fault detection output.
  • Install and configure the control plane: NVIDIA Base Command Manager including high-availability head nodes, operating system and cluster components, Slurm with Enroot and Pyxis, GPU and DOCA drivers, container toolkit and NCLI.
  • Set initial parameters for third-party storage and confirm throughput against the design.

Fabric *

  • InfiniBand: provisioning, high availability, partition keys for multi-tenancy, quality of service, adaptive routing, and link and bandwidth monitoring through UFM.
  • Or Spectrum-X Ethernet: RoCE, ECN and PFC configuration, quality of service, adaptive routing, telemetry.
  • Troubleshoot with the standard tooling - ibdiagnet, iblinkinfo, ib_write_lat and ib_write_bw, ibping, What Just Happened, UFM health.

Validation and acceptance - the half that is usually missing *

  • Single-node stress and burn-in.
  • HPL and single-node NCCL, including NVLink and NVLink Switch verification.
  • ClusterKit node assessment to find the weak node before the customer does.
  • NCCL east-west fabric bandwidth tests across the cluster.
  • Sustained burn-in using NCCL, HPL and NeMo workloads.
  • Storage throughput testing under realistic checkpoint load.
  • Produce the acceptance report: what was tested, what it returned, what was remediated.

Operate and remediate *

  • Diagnose and replace failed GPUs, network cards, fans and power supplies.
  • Tune storage and fabric where results fall short of specification.
  • Hand over to the customer or operations team with documentation that matches what was actually built.

Requirements

  • 5-8 years in data centre infrastructure engineering, of which at least 2-3 years hands-on with NVIDIA GPU hardware. This matches NVIDIA’s own stated prerequisite for its AI infrastructure certification.
  • Delivered at least one multi-node GPU cluster build. Node count, GPU model, fabric type and the candidate’s own scope must be stated, not implied.
  • Comfortable on a customer site, under programme pressure, working to someone else’s design and method of procedure.
  • Linux systems fluency; scripting for repeatable build steps.

Certifications - strongly preferred: *

  • NVIDIA-Certified Professional: AI Infrastructure (NCP-AII)
  • NVIDIA-Certified Professional: AI Networking (NCP-AIN) for fabric-leaning candidates

Benefits & conditions

  • Remote-only or lab-only experience. This work happens on site.
  • Data scientists, ML engineers and MLOps specialists.
  • Pure network engineers with no server, GPU or firmware exposure, unless hired explicitly against the fabric specialism.
  • Candidates who cannot name the clusters they built and what they personally did on them.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…