> Markdown version of [/jobs/ext/4693921-ml-performance-engineer-distributed-training](https://www.wearedevelopers.com/jobs/ext/4693921-ml-performance-engineer-distributed-training). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # ML Performance Engineer, Distributed Training - **Company:** ZETTABYTES, INC. - **Location:** Palo Alto, CA, United States - **Experience:** Expert - **Salary:** $210,000 - $270,000 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Software Applications, C++ (Programming Language), Cloud Computing, Computer Clusters, Profiling, Nvidia CUDA, Data Centers, Extract Transform Load (ETL), Distributed Computing Environment, Python (Programming Language), Open Source Technology, Graphics Processing Unit (GPU), Computer Network Operations, Pytorch, DeepSpeed, Parallel Computation, Machine Learning Operations, Model Inference, Megatron - **Published:** November 12, 2026 - **Apply:** https://www.careerbuilder.com/job-details/ml-performance-engineer-distributed-training-palo-alto-ca--03bf6d6f-d87d-4c80-8baf-d2d7e493a922 ## About the Role Performance Engineering, Performance Engineering, Cloud Computing, Artificial Intelligence (AI), GPU (Graphics Processing Unit), High Tech Industry, Computer Software, Network Operations Center, OEM (Original Equipment Manufacturer), Customer Support/Service, * 5+ years in ML systems, HPC or performance engineering * Hands-on experience training large models across many GPUs and nodes * Deep knowledge of PyTorch and at least one large-scale training framework * Strong understanding of collective communication libraries such as NCCL and of GPU interconnects * Experience with profiling tools such as Nsight Systems and PyTorch Profiler * Programming in Python and C++; CUDA experience is a strong plus Bonus Qualifications * Experience at a frontier lab, AI cloud or HPC centre * Contributions to open-source training frameworks * Familiarity with mixed-precision formats such as FP8 * Published work on large-scale training efficiency ## Description * AI computing company: we turn GPU hardware into reliable, high-utilisation AI compute through software, orchestration and full lifecycle operations * What we offer: enterprise GPU cloud and AI inference, software for data centre owners to run their own GPU clouds, turnkey and sovereign AI data centres, edge computing and GPU financing * Who we work with: GPU owners, data centre operators, enterprises, OEMs, integrators and financiers across multiple markets * How we work: fast-moving and high-accountability; we value speed, ambition, clarity, ownership and measurable outcomes Role Summary The ML Performance Engineer, Distributed Training helps customers and internal teams train large models efficiently on Zettabytes GPU clusters. The role analyses multi-node training jobs, finds where time is lost in compute, communication and data loading, and delivers changes that raise throughput and model FLOPs utilisation. The ML Performance Engineer works with infrastructure, networking and solutions teams, and often directly with customer research teams. Reporting Line Reports to: Co-CTO and CPO, * Performance analysis: profile distributed training jobs and identify bottlenecks across compute, communication, memory and input pipelines. * Parallelism strategies: advise on and tune data, tensor, pipeline and expert parallelism for large models. * Framework tuning: optimise training with PyTorch, Megatron-LM, DeepSpeed, FSDP and related frameworks. * Benchmarks: build and maintain training benchmarks that validate new clusters and track performance over time. * Customer engagements: support customers in onboarding and scaling training workloads on zCLOUD and zSUITE-managed clusters. * Cluster validation: work with infrastructure and network teams to confirm that new clusters meet training performance targets. * Checkpointing and data: improve checkpoint and data-loading performance with storage and platform teams. * Knowledge sharing: write guides and reference configurations for common model families and cluster sizes.