Principal Systems Engineer

NSCALE, LLC
San Francisco, United States of America
29 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
$ 225K

Job location

San Francisco, United States of America

Tech stack

Artificial Intelligence
Systems Engineering
Build Automation
Computer Clusters
Network Congestion
Software Debugging
Distributed Data Store
Ethernet
Firmware
InfiniBand
Network Layer
PCI Express
Performance Tuning
Remote Direct Memory Access
Ceph
Graphics Processing Unit (GPU)
High Performance Computing
Nvme

Job description

We are hiring a Principal Deployment Engineer to architect and lead the bringup of large-scale GPU clusters (hundreds to thousands of GPUs). This is a technical leadership role responsible for defining how we deploy, validate, and scale AI superclusters across sites.

You will own the full lifecycle of deployment-from rack design and fabric architecture to cluster validation frameworks and production readiness standards. You will set the bar for performance, reliability, and operational excellence.

This role combines deep hands-on expertise with system-level thinking and cross-functional leadership., Define the technical standards for node, rack, and full-cluster bringup. Lead large-scale GPU cluster deployments (multi-rack, multi-pod environments). Architect high-performance network fabrics (IB, RoCE, Ethernet) optimized for AI workloads. Establish cluster-level acceptance criteria and validation frameworks.

Performance & Fabric Architecture

Tune and validate NCCL, RDMA, GPUDirect, and collective operations at scale. Identify and eliminate performance bottlenecks across hardware, topology, and firmware layers. Drive congestion control and fabric optimization strategies. Define performance benchmarking methodology for AI training workloads.

Deployment Strategy & Scalability

Design repeatable deployment models for multi-site expansion. Build automation frameworks for provisioning and cluster validation. Establish deployment SLAs, quality gates, and operational readiness standards. Reduce time-to-capacity while increasing reliability.

Technical Leadership

Serve as the escalation point for complex bringup and performance issues. Mentor senior engineers and shape infrastructure best practices. Influence hardware selection, rack topology, and data center design decisions. Partner with executive leadership on infrastructure scaling strategy.

Requirements

Do you have experience in System performance optimization?, 10+ years of experience in large-scale infrastructure or HPC environments. Proven experience bringing up large GPU clusters (hundreds+ GPUs). Deep expertise in high-speed networking (InfiniBand, RoCE, Ethernet fabrics). Strong understanding of server architecture (PCIe, NUMA, memory hierarchy). Experience debugging performance issues across compute and network layers. Strong automation and systems-level thinking.

Strongly Preferred

Experience scaling AI training clusters for frontier models. Experience with liquid cooling or ultra-high-density deployments. Knowledge of distributed storage systems (Lustre, Ceph, NVMe-oF). Experience defining infrastructure standards in a fast-growing organization.

About the company

We are building AI infrastructure for frontier-scale workloads. Our platform is designed for high-density, high-performance GPU clusters that push the limits of power, networking, and distributed compute. As a startup, we move fast, operate with ownership, and expect technical leaders to define standards-not just follow them., Superclusters are brought online quickly, predictably, and at peak performance. Deployment processes scale from first cluster to multi-site expansion. Infrastructure becomes a competitive advantage. * You define the technical blueprint for how we scale AI infrastructure.

Apply for this position