> Markdown version of [/jobs/ext/2037250-software-engineer-ai-infrastructure](https://www.wearedevelopers.com/jobs/ext/2037250-software-engineer-ai-infrastructure). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineer, AI Infrastructure - **Company:** Tesla Motors - **Location:** Palo Alto, CA, United States - **Experience:** Experienced - **Salary:** $168,000.0 - $252,000.0 - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Bash Shell, Computer Clusters, Configuration Management, Nvidia CUDA, Data Centers, Software Debugging, Linux, General Parallel File Systems, InfiniBand, Python (Programming Language), Remote Direct Memory Access, Ansible, Prometheus, Software Engineering, AI Infrastructure, Data Logging, Graphics Processing Unit (GPU), Grafana, Kubernetes, Slurm, Machine Learning Operations - **Published:** August 12, 2026 - **Apply:** https://jobs.localjobnetwork.com/apply/add/85100354/1 ## About the Role * Strong software engineering instincts applied to infrastructure: you write maintainable automation and services, not only operate tickets * Proficiency in Python and/or Bash (additional languages a plus); comfort building tools others will run in production * Deep Linux experience in production (boot/image or config management, networking stack, performance debugging, package/driver realities on real hardware) * GPU cluster operations or AI training infrastructure (NVIDIA stack, driver/CUDA ecosystem familiarity, multi-node job behavior) * HPC schedulers and workload managers (Slurm, LSF, or similar); exposure to Kubernetes for ML/infra workloads a plus * High-performance networking: RDMA, RoCEv2, InfiniBand, congestion behavior, and host-side tuning for collective-heavy training * Parallel / high-performance storage (Lustre, GPFS/Spectrum Scale, BeeGFS, or cloud/parallel equivalents) and its failure modes under training I/O * Experience with configuration management / IaC-style automation at fleet scale (e.g. Ansible, or equivalent) * Willingness to participate in a 24O7 on-call rotation for critical AI training infrastructure (cadence shared during hiring; escalation and automation are first-class goals of the role) * Degree in CS, EE, or related field, or equivalent practical experience ## Description * Keep the training fleet up; Drive reliability and efficiency of GPU clusters so research and production training progress is not gated on infrastructure toil or silent degradation * Ship automation as product; Replace manual ops with reliable, testable software, APIs, job pipelines, operators/controllers, and libraries other engineers and on-call can depend on * Own observability and remediation paths; Build metrics, logging, alerting, and automated remediation (e.g. Prometheus/Grafana-based stacks) that detect bad nodes, sick GPUs, fabric/storage faults, and scheduler failures and fence or repair safely at scale * Raise training reliability through software; Model failure domains, reduce blast radius of HW/SW faults on long-running multi-node jobs, and improve effective capacity (less idle accelerators, fewer silent stragglers) * Develop performance-oriented systems tooling; Build and extend tools that measure and improve throughput/stability across OS, network, storage, and runtime layers for real distributed training workloads * Integrate with scheduling and workload platforms; Engineer against and improve HPC/AI scheduling and orchestration surfaces (e.g. Slurm/LSF and related platforms; containers/K8s where used) so capacity is allocatable, debuggable, and efficient * Partner across the stack working with ML infrastructure users, networking, data center, and hardware/vendor partners to bring up capacity and expose clean software interfaces, not ad-hoc runbooks alone * Participate in a 24O7 rotation for critical training infra; invest in SLIs/SLOs, runbooks-as-code, and permanent fixes that drive pages down over time ## Related Videos - [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes) - [Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps](https://www.wearedevelopers.com/videos/1894-shipping-faster-with-less-render-on-cloud-hosting-ai-workloads-and-the-future-of-devops) - [5 steps for running a Kubernetes environment at scale](https://www.wearedevelopers.com/videos/88-5-steps-for-running-a-kubernetes-environment-at-scale) - [Docker network without Docker](https://www.wearedevelopers.com/videos/1418-docker-network-without-docker) - [Developing an AI.SDK](https://www.wearedevelopers.com/videos/198-developing-an-ai-sdk) - [All your telemetry data from any source in one place](https://www.wearedevelopers.com/videos/57-all-your-telemetry-data-from-any-source-in-one-place) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)