> Markdown version of [/jobs/48411-gpu-cluster-engineer-systems-platform](https://www.wearedevelopers.com/jobs/48411-gpu-cluster-engineer-systems-platform). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # GPU Cluster Engineer, Systems & Platform - **Company:** Sciforium - **Location:** San Francisco, United States - **Experience:** Expert - **Salary:** €150,000.0 - €220,000.0 - **Skills:** Kubernetes - **Published:** August 21, 2026 - **Apply:** https://jobs.ashbyhq.com/Sciforium/62d7ad5e-d00b-4e6d-b5be-b02ea9ef65ea ## About the Role Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications. ## **About the Role** We are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure. **Key Responsibilities** - **OS Bring-Up & Node Lifecycle Engineering** - Golden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready. - Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool. - Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver ↔ CUDA/ROCm ↔ framework compatibility matrix across the fleet. - Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA. - **Configuration Management & Automation** - Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment. - Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted. - Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation. - **Orchestration & Scheduling (Kubernetes & Slurm)** - Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate. - Training Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis). - Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible. - **GPU Driver & ML Stack Engineering** - Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA). - Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services. - Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions. - **Advanced Debugging & Observability** - Escalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops. - Observability: Own software-layer monitoring (DCGM exporter, Prometheus/Grafana, alerting) plus job-level GPU utilization and cluster efficiency reporting. ## Description **Qualifications** - **Must-Haves:** - 5+ years in systems/infrastructure engineering with significant GPU cluster, HPC, or large-scale ML infrastructure experience. - Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field. - Deep Linux internals expertise: kernel modules/DKMS, systemd, cgroups, NUMA, and system performance tuning. - Hands-on experience with NVIDIA (CUDA) and/or AMD (ROCm) driver and runtime stacks on modern accelerators (H200/B200, MI325x/MI355x class), including kernel-level debugging. - Production Kubernetes experience with GPU workloads, plus working knowledge of HPC schedulers (Slurm/Run:AI) — or the reverse (deep Slurm, working K8s). - Strong configuration management experience (Ansible or SaltStack) with Git-based, code-reviewed infrastructure workflows. - Provisioning and image tooling experience (Packer, MaaS, Foreman, Terraform, or similar) for automated, reproducible node builds. - Client-side experience with distributed filesystems (Lustre, GPFS, Weka) and checkpoint I/O optimization. - Container fluency: Docker/containerd and the NVIDIA Container Toolkit or ROCm equivalent. - Proficiency in Python and Bash for automation and tooling. - Working knowledge of NCCL and RDMA networking (InfiniBand/RoCE, GPUDirect) and of PyTorch/JAX runtime behavior. - **Nice-to-Haves:** - Experience directly supporting foundation model training teams — multi-node job failure debugging, checkpoint pipeline tuning, and framework-level performance triage — ideally in a startup or research-heavy environment. - Experience deploying and tuning inference/serving stacks (vLLM, Triton Inference Server, TensorRT-LLM) for latency and throughput targets. - GPU/system profiling tools: Nsight Systems/Compute, rocprof, perf, eBPF. ## About Sciforium [Company profile](https://www.wearedevelopers.com/companies/4292-sciforium) ### More Jobs at Sciforium - [LLM Training Engineer](https://www.wearedevelopers.com/jobs/48420-llm-training-engineer) - [LLM Dataset Engineer](https://www.wearedevelopers.com/jobs/48419-llm-dataset-engineer) - [Model Implementation Engineer](https://www.wearedevelopers.com/jobs/48421-model-implementation-engineer) - [Senior AI Serving Engineer, Backend](https://www.wearedevelopers.com/jobs/48414-senior-ai-serving-engineer-backend) - [ML Engineer](https://www.wearedevelopers.com/jobs/48422-ml-engineer) ## Related Videos - [Understanding Kubernetes in a visual way](https://www.wearedevelopers.com/videos/100085-understanding-kubernetes-in-a-visual-way) - [The Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)](https://www.wearedevelopers.com/videos/2067-the-gashlycrumb-tinies-of-ai-networking-you-must-know-or-languish) - [Instant KAI Sandboxes with vCluster: Multi-Tenant, Multi-Scheduler GPU Sharing](https://www.wearedevelopers.com/videos/100333-instant-kai-sandboxes-with-vcluster-multi-tenant-multi-scheduler-gpu-sharing) - [A Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast Training and Inference on Kubernetes](https://www.wearedevelopers.com/videos/1625-a-deep-dive-on-how-to-leverage-the-nvidia-gb200-for-ultra-fast-training-and-inference-on-kubernetes) - [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes) - [Microservices: how to get started with Spring Boot and Kubernetes](https://www.wearedevelopers.com/videos/242-microservices-how-to-get-started-with-spring-boot-and-kubernetes) ## Related Articles - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [7 Cloud Computing Trends Coming in 2025 for Developers](https://www.wearedevelopers.com/magazine/412-7-cloud-computing-trends-coming-in-2025-for-developers) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)