> Markdown version of [/jobs/ext/1196714-ml-infrastructure-engineer](https://www.wearedevelopers.com/jobs/ext/1196714-ml-infrastructure-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # ML Infrastructure Engineer - **Company:** White Circle - **Location:** London, UK - **Salary:** £180,000.0 - £350,000.0 - **Contract:** Permanent contract - **Skills:** Training Data, Artificial Intelligence, C++ (Programming Language), Computer Clusters, Profiling, Nvidia CUDA, Data Control, Extract Transform Load (ETL), Software Debugging, InfiniBand, Python (Programming Language), Multiprocessing, Performance Tuning, Remote Direct Memory Access, Rust (Programming Language), Data Processing, Graphics Processing Unit (GPU), Pytorch, Large Language Models, Multi-Agent Systems, Concurrency, Deep Learning, Perf (Linux), Kubernetes, Slurm, Machine Learning Operations, TensorRT, Asynchronous Programming, Golang - **Published:** July 7, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=246700f98be4f360 ## About the Role * Have designed, built, or maintained distributed RL/post-training systems at scale and are fluent in their moving parts: rollouts, replay buffers, reward signals, data filtering, policy updates, evaluation loops, and failure analysis * Are familiar with deep learning frameworks such as PyTorch or JAX * Are proficient in Python, including concurrency, asynchronous programming, multiprocessing, and performance optimization * Can debug distributed GPU workloads across CUDA runtime, container runtime, driver versions, NCCL or equivalent communication layers, networking, storage, scheduling, and checkpointing * Have experience with profiling tools across the stack, for example py-spy, PyTorch profiler, Nsight, perf, tracing, metrics, logs, or custom instrumentation * Have experience with inference stacks such as vLLM, SGLang, TensorRT-LLM, Dynamo, or custom serving infrastructure * Can reason from system metrics back to model behavior: when latency, queueing, sampling, data order, rollout throughput, or infrastructure failures affect learning * Have a strong ownership mindset: you can take an ambiguous infrastructure problem, make it concrete, ship a working system, and improve it from real feedback, * Experience in a high-bar AI infra, research, or model environment such as xAI/Grok, Qwen, ByteDance AI infra/research, Prime Intellect, or similar teams * Custom training framework support or ownership: distributed training, fine-tuning pipelines, trainers, schedulers, checkpointing, data loaders, model/eval integration, or performance tooling * Serious use of Claude Code, Codex, Kimi Code, Pi Agent, Droid, or similar agentic coding systems as a development surface * Experience with GPU clusters on Kubernetes, Slurm, Ray, custom schedulers, or cloud GPU orchestration * NCCL, UCX, NVSHMEM, RDMA, InfiniBand, RoCE, or EFA * Rust, C++, CUDA, Go, or systems-level performance work ## Description TLDR: We are looking for an ML Infrastructure Engineer to build the systems behind our LLM post-training, RL, evaluation, inference, and agentic development workflows. You will work close to researchers, GPUs, training loops, data control systems, evals, inference stacks, and the infrastructure decisions that directly affect model learning and product quality., * Build robust, flexible, and scalable RL and post-training pipelines, including smoke tuning runs for quality testing and approach ablations * Design data control systems that govern what the model sees, when it sees it, and how training data flows through rollouts, replay, filtering, evaluation, and policy updates * Tune training and inference end-to-end for high throughput across the systems that matter: networking, memory, compute scheduling, data loading, storage, checkpointing, and I/O * Investigate how infrastructure choices affect learning dynamics, eval quality, model behavior, and training stability - staying close to the state of the art in LLMs, RL, and post-training * Build infrastructure for model iteration: experiment runs, artifacts, evals, dashboards, failure inspection, reproducibility, and cost visibility * Work on inference infrastructure where it affects post-training and evaluation loops * Build and improve agentic development environments: coding-agent harnesses, browser/tool integrations, terminal/runtime sandboxes, repo-aware workflows, and multi-agent orchestration * Work closely with the team: plan future steps, discuss tradeoffs, share context early, and stay in touch while building ## Related Videos - [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes) - [Go with the Flow: Stop the Leaks Before Your Memory's a Waterfall!](https://www.wearedevelopers.com/videos/100073-go-with-the-flow-stop-the-leaks-before-your-memory-s-a-waterfall) - [Photonic Computing: Programming a New Class of AI Accelerators (incl. Live Coding)](https://www.wearedevelopers.com/videos/100196-photonic-computing-programming-a-new-class-of-ai-accelerators-incl-live-coding) - [LLMOps-driven fine-tuning, evaluation, and inference with NVIDIA NIM & NeMo Microservices](https://www.wearedevelopers.com/videos/1582-llmops-driven-fine-tuning-evaluation-and-inference-with-nvidia-nim-nemo-microservices) - [Retooling and refactoring - an investment in people.](https://www.wearedevelopers.com/videos/371-retooling-and-refactoring-an-investment-in-people) - [Efficient deployment and inference of GPU-accelerated LLMs​](https://www.wearedevelopers.com/videos/929-efficient-deployment-and-inference-of-gpu-accelerated-llms) ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)