> Markdown version of [/jobs/ext/2599564-ai-infrastructure-engineer](https://www.wearedevelopers.com/jobs/ext/2599564-ai-infrastructure-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Infrastructure Engineer

- **Company:** Sciforium Corporation
- **Location:** San Francisco, CA, United States
- **Experience:** Expert
- **Salary:** $150,000.0 - $220,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Bash Shell, Computer Clusters, Configuration Management, Nvidia CUDA, Computer Engineering, Software Debugging, General Parallel File Systems, InfiniBand, Python (Programming Language), Linux Kernel, Node.Js, Performance Tuning, Remote Direct Memory Access, Ansible, Tensorflow, Prometheus, Memory Leaks, Weka, AI Infrastructure, Pytorch, Saltstack, Large Language Models, Grafana, Git, Kubernetes, Information Technology, Low Latency, Slurm, Machine Learning Operations, TensorRT, Terraform, Software Version Control, Docker
- **Published:** August 5, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=10322f033b271931

## About the Role

+ 5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience.
 + Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
 + Deep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning.
 + Hands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging.
 + Production Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) - or the reverse (deep Slurm, working K8s).
 + Strong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows.
 + Provisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds.
 + Client-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization.
 + Container fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent.
 + Proficiency in Python and Bash for automation and tooling.
 + Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior.
 * Nice-to-Haves:
 + Experience directly supporting foundation model training teams - multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage - ideally in a startup or research-heavy environment.
 + Experience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets.
 + GPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF.

## Description

We are looking for an AI Infrastructure Engineer to own the entire software stack of our GPU clusters - from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups - our foundation model training teams and our model serving/product teams - ensuring both run on correctly configured, well-managed, high-performance infrastructure., * OS Bring-Up & Node Lifecycle Engineering

 + Golden Images & Automated Bring-Up: Own the node software definition - versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks - and the automated pipeline that takes a node from base OS to production-ready.
 + Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.
 + Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver CUDA/ROCm framework compatibility matrix across the fleet.
 + Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.
 * Configuration Management & Automation
 + Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.
 + Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.
 + Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.
 * Orchestration & Scheduling (Kubernetes & Slurm)
 + Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads - NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.
 + Training Schedulers: Operate Slurm (or Run:AI) for multi-node training - partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).
 + Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.
 * GPU Driver & ML Stack Engineering
 + Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack - NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) - including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).
 + Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.
 + Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.
 * Advanced Debugging & Observability
 + Escalation Point: Own the hard problems - NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.
 + Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting.


## Related Videos

- [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [The Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)](https://www.wearedevelopers.com/videos/2067-the-gashlycrumb-tinies-of-ai-networking-you-must-know-or-languish)
- [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews)
- [Stop Committing Your Secrets - GIt Hooks To The Rescue!](https://www.wearedevelopers.com/videos/573-stop-committing-your-secrets-git-hooks-to-the-rescue)
- [AI Factories at Scale](https://www.wearedevelopers.com/videos/1139-ai-factories-at-scale)

## Related Articles

- [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)
- [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)