> Markdown version of [/jobs/ext/2709182-ml-training-infrastructure-engineer](https://www.wearedevelopers.com/jobs/ext/2709182-ml-training-infrastructure-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# ML Training Infrastructure Engineer

- **Company:** Dyna Robotics
- **Location:** Redwood City, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Computer Clusters, Compilers, Program Optimization, Profiling, Distributed Computing Environment, Distributed Systems, Memory Management, Protocol Buffers, Job Scheduling, Data Processing, Graphics Processing Unit (GPU), Pytorch, Kubernetes, Slurm, Machine Learning Operations, TensorRT
- **Published:** September 4, 2026
- **Apply:** https://startup.jobs/ml-infrastructure-engineer-training-dyna-robotics-8139937

## About the Role

* 7+ Years of Engineering: With a track record of leading technical projects in high-performance computing (HPC) or ML infrastructure.
 * ML Systems Mastery: Deep experience with PyTorch and distributed training frameworks (DeepSpeed, Accelerate). You understand the nuances of mixed precision and gradient accumulation.
 * Infrastructure Expertise: Hands-on experience managing cloud GPU environments (GCP/AWS) and container orchestration (Kubernetes).
 * Low-Level Intuition: A fundamental understanding of distributed systems, including race conditions, memory management, and NCCL/inter-node communication.
 * Ownership Mindset: You don't just "deploy" code; you design, build, and operate systems end-to-end to unblock fast-moving research.

Bonus Points For

 * Experience with Robotics Data Formats (MCAP, Protobuf) or multimodal models (VLAs).
 * Deep ML systems experience: custom kernels (Triton), compilers, or runtime optimization.
 * Experience as a founding or early-stage infrastructure hire.

## Description

As a ML Training Infrastructure Engineer, you will architect and build the systems that turn our multi-cloud GPU fleet into a training engine our researchers love. Your charter is singular and broad: own training infrastructure end-to-end so that every GPU is busy, every run is reproducible, and every researcher's next experiment is one command away., * Scale Distributed Training: Architect and own the infrastructure for large-scale GPU clusters. You'll implement sharding, activation checkpointing, and memory optimization (ZeRO, FSDP) to enable the training of massive multimodal models.

 * Optimize Researcher Ergonomics: Build a research codebase and job scheduling system (Kubernetes/SLURM) that prioritizes fast iteration, automated retries, and seamless failure recovery.
 * High-Performance Data Handling: Design high-throughput pipelines to ingest and transform terabytes of multimodal robot data (video, proprioception, 3D signals), ensuring dataloaders never starve the GPUs.
 * Production Inference: Build low-latency inference pipelines for real-time robot control. You'll apply quantization, distillation, and model compilation (TensorRT, Triton) to move models from the lab to the physical world.
 * Deep Systems Profiling: Dive into the weeds of GPU utilization, I/O bottlenecks, and memory fragmentation to squeeze every bit of performance out of our expanding compute fleet.


## Related Videos

- [Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated](https://www.wearedevelopers.com/videos/100099-tour-de-force-open-source-llm-inference-optimization-from-simple-to-sophisticated)
- [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes)
- [Introduction to Azure Machine Learning](https://www.wearedevelopers.com/videos/368-introduction-to-azure-machine-learning)
- [Photonic Computing: Programming a New Class of AI Accelerators (incl. Live Coding)](https://www.wearedevelopers.com/videos/100196-photonic-computing-programming-a-new-class-of-ai-accelerators-incl-live-coding)
- [Efficient deployment and inference of GPU-accelerated LLMs​](https://www.wearedevelopers.com/videos/929-efficient-deployment-and-inference-of-gpu-accelerated-llms)
- [Serverless deployment of (large) NLP models ](https://www.wearedevelopers.com/videos/158-serverless-deployment-of-large-nlp-models)

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)