ML Performance Engineer, Distributed Training

• AI computing company: we turn GPU hardware into reliable, high-utilisation AI compute through software, orchestration and full lifecycle operations
• What we offer: enterprise GPU cloud and AI inference, software for data centre owners to run their own GPU clouds, turnkey and sovereign AI data centres, edge computing and GPU financing
• Who we work with: GPU owners, data centre operators, enterprises, OEMs, integrators and financiers across multiple markets
• How we work: fast-moving and high-accountability; we value speed, ambition, clarity, ownership and measurable outcomes

Role Summary

The ML Performance Engineer, Distributed Training helps customers and internal teams train large models efficiently on Zettabytes GPU clusters.

The role analyses multi-node training jobs, finds where time is lost in compute, communication and data loading, and delivers changes that raise throughput and model FLOPs utilisation.

The ML Performance Engineer works with infrastructure, networking and solutions teams, and often directly with customer research teams.

Reporting Line

Reports to: Co-CTO and CPO, * Performance analysis: profile distributed training jobs and identify bottlenecks across compute, communication, memory and input pipelines.

• Parallelism strategies: advise on and tune data, tensor, pipeline and expert parallelism for large models.
• Framework tuning: optimise training with PyTorch, Megatron-LM, DeepSpeed, FSDP and related frameworks.
• Benchmarks: build and maintain training benchmarks that validate new clusters and track performance over time.
• Customer engagements: support customers in onboarding and scaling training workloads on zCLOUD and zSUITE-managed clusters.
• Cluster validation: work with infrastructure and network teams to confirm that new clusters meet training performance targets.
• Checkpointing and data: improve checkpoint and data-loading performance with storage and platform teams.
• Knowledge sharing: write guides and reference configurations for common model families and cluster sizes.

Performance Engineering, Performance Engineering, Cloud Computing, Artificial Intelligence (AI), GPU (Graphics Processing Unit), High Tech Industry, Computer Software, Network Operations Center, OEM (Original Equipment Manufacturer), Customer Support/Service, * 5+ years in ML systems, HPC or performance engineering

• Hands-on experience training large models across many GPUs and nodes
• Deep knowledge of PyTorch and at least one large-scale training framework
• Strong understanding of collective communication libraries such as NCCL and of GPU interconnects
• Experience with profiling tools such as Nsight Systems and PyTorch Profiler
• Programming in Python and C++; CUDA experience is a strong plus

Bonus Qualifications

• Experience at a frontier lab, AI cloud or HPC centre
• Contributions to open-source training frameworks
• Familiarity with mixed-precision formats such as FP8
• Published work on large-scale training efficiency

• A training benchmark suite validates every new cluster before handover
• Reference configurations exist for the most common model sizes
• At least two customer training workloads show measured throughput gains
• Performance findings feed into infrastructure and network designs

Why This Role Matters

• Customer outcomes: training efficiency decides how fast customers reach results and how much they spend.
• Platform proof: strong training performance shows what Zettabytes clusters can do.

Working Norms at Zettabyte

• Bias to action: deliver drafts quickly, then iterate on feedback
• Be precise: double-check numbers and facts; sloppy analysis creates downstream cost
• Make it usable: package outputs for decision-making, not data dumps
• Own the process: whoever drives a deliverable manages stakeholders and timelines
• Learn the ecosystem: understand GPUs, cluster operations and the buyer landscape, * Compensation: USD 210,000 to 270,000 a year base salary, plus equity
• Benefits: Medical, dental and vision insurance; 401(k) retirement plan; paid time off and US public holidays; equity
• Work authorisation: Required in the United States
• Equal opportunity: Zettabyte is an equal opportunity employer; qualified applicants with arrest or conviction records will be considered in line with the California Fair Chance Act

Zettabyte is a supplier of the United Nations Global Marketplace (UNGM #1117361) and is committed to aiding the United Nations in achieving its Sustainable Development Goals.