> Markdown version of [/jobs/ext/4113088-principal-ai-infrastructure-engineer](https://www.wearedevelopers.com/jobs/ext/4113088-principal-ai-infrastructure-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal AI Infrastructure Engineer - **Company:** HIGH RISES, LLC - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Bash Shell, BIOS, Cloud Computing, Configuration Management, System Configuration, Data Centers, Linux, Distributed Data Store, Ethernet, Firmware, InfiniBand, Python (Programming Language), Node.Js, PCI Express, Performance Tuning, Remote Direct Memory Access, Ansible, Systems Integration, AI Infrastructure, Network Switches, Performance Testing, Kubernetes, Information Technology, Bare Metal, Slurm, 3-tier Architectures - **Published:** October 15, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=2202f881cc205daa ## About the Role * Bachelor's degree in computer science, engineering, or a related field, or equivalent practical experience. * 8+ years of hands-on Linux systems engineering, HPC, cloud, or large-scale infrastructure experience, including senior-level troubleshooting. * Proven experience bringing GPU or HPC clusters into production and qualifying systems against performance and reliability targets. * Strong expertise in Linux, automation (Python/Bash/Ansible), bare-metal provisioning, configuration management, and systems diagnostics. * Hands-on experience with NVIDIA GPU platforms, NCCL, Field Diagnostics, and multi-node performance testing. * Practical experience with Kubernetes and Slurm; familiarity with high-speed Ethernet, InfiniBand/RDMA, and parallel or distributed storage. Core Competencies * Deep technical curiosity and the ability to isolate difficult problems across hardware, firmware, operating systems, and distributed workloads. * Strong ownership of outcomes, with a systematic approach to testing, documentation, root cause analysis, and continuous improvement. * Comfortable collaborating with highly experienced peer engineers while driving independent technical decisions and execution. * Clear communication with engineering leadership, vendors, site personnel, and customer-facing stakeholders. ## Description We are seeking a senior, hands-on AI Cluster Integration Engineer to bring large-scale GPU compute clusters from physical installation to validated, commercially ready production. Working alongside a team of senior infrastructure engineers, this person will own critical elements of cluster bring-up, bare-metal provisioning, advanced systems configuration, performance optimization, and acceptance testing. After deployment, the engineer will provide Tier 3 technical escalation and sustained performance support for operating clusters. The focus is on compute-platform readiness and workload performance rather than primary ownership of network switch and fabric configuration., Cluster Bring-Up and Systems Integration * Lead technical bring-up of materially complete, racked, powered, and cabled AI clusters through production readiness and customer handoff. * Establish and validate system baselines across BIOS/BMC, firmware, GPU drivers, operating systems, and cluster software stacks. * Deploy and optimize automated bare-metal provisioning, imaging, node configuration, and lifecycle workflows at scale. * Configure and troubleshoot advanced Linux and GPU systems, including kernel, driver, accelerator, and host-level performance settings. * Integrate and validate Kubernetes and Slurm environments in partnership with platform, networking, and operations engineers. Performance Engineering and Acceptance * Execute NVIDIA Field Diagnostics, L11/L12 validation, burn-in, stability, and hardware health qualification procedures. * Run and analyze NCCL collective communication tests, GPU benchmarks, and distributed workload performance at node, rack, and cluster scale. * Troubleshoot GPU, NVLink/NVSwitch, PCIe, RDMA, and InfiniBand/Ethernet performance issues, partnering with network engineers on fabric-related remediation. * Identify configuration bottlenecks and optimize throughput, latency, scaling efficiency, and cluster reliability. * Own technical acceptance criteria, test evidence, defect tracking, remediation verification, and readiness recommendations for commercial service. Tier 3 Operations and Reliability * Serve as a senior escalation point for complex production incidents involving compute nodes, GPU software, provisioning, scheduling, and degraded performance. * Perform deep root cause analysis, coordinate vendor engineering escalations, and lead corrective actions for recurring or systemic issues. * Support fleet-level upgrades, configuration changes, cluster expansions, and post-maintenance validation with appropriate change controls. * Develop diagnostic tooling, automation, runbooks, and operational handoff material for SRE, NOC, and site support teams. Cross-Functional Execution * Work closely with infrastructure engineering, site operations, OEMs, and software/platform teams to resolve cross-stack issues and meet delivery milestones. * Provide concise technical status, benchmark findings, risks, and acceptance recommendations to engineering leadership and stakeholders., This is a full-time, engineering role supporting large-scale AI compute infrastructure across multiple sites. Travel to data centers for cluster commissioning, acceptance, and complex troubleshooting will be required. Flexibility to assist with critical incidents or maintenance activities outside regular business hours is expected. This role is hybrid or remote depending on location. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.