> Markdown version of [/jobs/ext/2721005-senior-ai-infrastructure-engineer-physical-infrastructure](https://www.wearedevelopers.com/jobs/ext/2721005-senior-ai-infrastructure-engineer-physical-infrastructure). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior AI Infrastructure Engineer, Physical Infrastructure - **Company:** Anduril Industries - **Location:** Costa Mesa, CA, United States - **Experience:** Expert - **Salary:** $166,000.0 - $220,000.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, BIOS, Network Congestion, Software Debugging, Fault Tolerance, Firmware, InfiniBand, Azure Machine Learning, Weka, AI Infrastructure, Datadog, Graphics Processing Unit (GPU), Large Language Models, Kubernetes, Low Latency, Deployment Automation - **Published:** September 4, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9144148/senior-ai-infrastructure-engineer-physical-infrastructure ## About the Role * 10+ years in a hands on infrastructure, HPC, or datacenter engineering role supporting GPU compute at scale. * Hands on experience with H200/B200/B300 (or comparable) GPU systems: bring up, cabling, firmware/driver management. * Experience with high performance interconnects (NVLink, InfiniBand, RoCE, Spectrum-X) in clusters of hundreds of GPUs. * Experience with high performance parallel storage (VAST, DDN, Weka, Lustre, or similar). * Kubernetes required; Run:ai or similar GPU scheduling/orchestration experience strongly preferred. * Strong automation background. You build repeatable, automated deployment pipelines rather than manual processes. * Able to lift/move 50+ lbs and perform physical datacenter work (rack/stack/cable/troubleshoot). * Eligible to obtain and maintain an active U.S. Top Secret clearance. PREFERRED QUALIFICATIONS * Experience with NVIDIA NVL72 rack scale systems. * Experience supporting LLM token serving/inference infrastructure alongside training clusters. * Network fabric tuning experience (congestion control, adaptive routing, QoS) for RoCE/InfiniBand at scale. * Familiarity with GPU/network observability tooling (DCGM, fabric telemetry) and automated fault detection. * Experience supporting infrastructure as a shared platform serving multiple internal customer teams with differing requirements. ## Description We're looking for a Senior AI Infrastructure Engineer to lead the vision, execution, and long-term stability of how Anduril trains with GPUs at scale. In this role, you will take absolute ownership of cluster robustness, ensuring our high-performance GPU systems are highly available, fault-tolerant, and resilient for ML platform and research teams company-wide. This is a highly hands-on role where your primary focus is logical stability and automated resilience-building self-healing mechanisms to proactively detect and isolate hardware faults, tuning NCCL and high-speed networking, and optimizing Kubernetes, Run:AI, and Ray scheduling. By replacing manual triage with automated deployment tooling and deep observability, you will ensure our massive-scale training infrastructure runs seamlessly and scales without linear headcount growth. WHAT YOU'LL DO * Rack, stack, cable, and bring up GPU compute (H200/B200/B300, NVL72) including physical topology, power, cooling, firmware/BIOS, and burn in validation. * Build and tune the interconnect fabric (NVLink, InfiniBand, RoCE, Spectrum-X) connecting hundreds of GPUs into low latency training and inference clusters. * Integrate high performance parallel storage (VAST, DDN, Weka) to sustain the throughput demanded by distributed training and terabyte scale multi modal datasets across Anduril's programs. * Automate cluster deployment and configuration end to end, including infrastructure as code for bring up, firmware/driver management, and fabric config, so new capacity comes online with minimal manual work. * Operate and extend our Kubernetes/Run:AI environment for GPU scheduling, quota management, and multi tenant workload isolation across research and engineering teams company wide. * Own fleet health: monitoring, alerting, and rapid triage of hardware and network faults (bad transceivers, GPU Xid errors, NCCL/collective failures, RoCE congestion). * Onboard engineers and researchers onto the platform and act as their escalation point, working directly alongside them to debug, train, and optimize their workloads whenever infrastructure, not the model, is the bottleneck. * Partner with product facing teams across Anduril to understand emerging compute needs and translate them into platform capability., To ensure your safety and help you navigate your job search with confidence, please keep the following critical points in mind: * No Financial Requests: Anduril will never solicit payment or demand personal financial details (such as banking information, credit card numbers, or social security numbers) at any stage of our hiring process. Our legitimate recruitment is entirely free for candidates. ## Related Videos - [10M Data Records Lost, Underwater Computing, and Psychedelic Fish - Matthias Geniar](https://www.wearedevelopers.com/videos/1908-10m-data-records-lost-underwater-computing-and-psychedelic-fish-matthias-geniar) - [Debugging in the Dark](https://www.wearedevelopers.com/videos/1658-debugging-in-the-dark) - [AI Factories at Scale](https://www.wearedevelopers.com/videos/1139-ai-factories-at-scale) - [Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure](https://www.wearedevelopers.com/videos/1583-software-engineering-social-connection-yubo-s-lean-approach-to-scaling-an-80m-user-infrastructure) - [Building the Nervous System of AI - Michael Kagan (NVIDIA)](https://www.wearedevelopers.com/videos/2133-building-the-nervous-system-of-ai-michael-kagan-nvidia) - [How to Submit CFPs and Get into Public Speaking - Moran Weber](https://www.wearedevelopers.com/videos/2112-how-to-submit-cfps-and-get-into-public-speaking-moran-weber) ## Related Articles - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)