> Markdown version of [/jobs/ext/1447821-senior-software-engineer-i-ai-inference-data-plane](https://www.wearedevelopers.com/jobs/ext/1447821-senior-software-engineer-i-ai-inference-data-plane). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Software Engineer I - AI Inference Data Plane - **Company:** DigitalOcean, LLC - **Location:** San Francisco, CA, United States (Remote available) - **Experience:** Expert - **Salary:** $139,200.0 - $174,000.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Cloud Computing, Python (Programming Language), Open Source Technology, Performance Tuning, Digitalocean, Software Engineering, Systems Integration, User-Centered Design, Load Balancing, Large Language Models, Kubernetes, Optimization Algorithms, Free and Open-Source Software, TensorRT, Golang - **Published:** July 26, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=d8282f908c624221 ## About the Role * AI/ML Domain Knowledge: Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or TensorRT. * Inference Frameworks: Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve. * Inference Engine Depth: Hands-on experience with vLLM or alternatives (SGLang, TensorRT-LLM, TGI, Modular MAX), including internals like continuous batching, paged attention, and prefix caching. * Distributed Inference Fluency: Understanding of why cluster-scale serving is hard: KV-cache locality is partitioned across workers, naive round-robin routing destroys cache hit rates and tail latency, and disaggregated prefill/decode requires fast cross-pod KV transfer (e.g., NIXL). * Upstream Track Record: Merged contributions to vLLM, llm-d, SGLang, or similar projects strongly preferred. * Architecture Proficiency: Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization). * Software Engineering: Expert-level proficiency in GoLang or Python and familiarity with gRPC. * Cloud Operations: Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean. * Open Source Mindset: Experience integrating and building with open-source software. ## Description * Technical Leadership: Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models. * System Design: Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards. * Performance Optimization: Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing. * Collaboration: Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs. * Distributed Serving at Scale: Build on Kubernetes-native distributed inference frameworks like llm-d (or alternatives such as NVIDIA Dynamo, Ray Serve, KServe) to deliver prefill/decode disaggregation, KV-cache-aware routing, tiered prefix caching, and wide expert parallelism for MoE models. * Flow Control & Load Balancing: Solve the distributed-systems problems unique to LLM serving - inference-aware load balancing on queue depth, cache locality, and predicted latency; flow control and fairness across tenants; autoscaling inference pools; and moving gigabytes of KV-cache between prefill and decode instances with negligible overhead. * Open Source Contributions: Contribute upstream to llm-d, vLLM, and the inference gateway ecosystem, and represent DigitalOcean in these communities. * Mentorship: Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement. * Operational Excellence: Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health. ## Related Videos - [Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated](https://www.wearedevelopers.com/videos/100099-tour-de-force-open-source-llm-inference-optimization-from-simple-to-sophisticated) - [Understanding Kubernetes in a visual way](https://www.wearedevelopers.com/videos/100085-understanding-kubernetes-in-a-visual-way) - [Go with the Flow: Stop the Leaks Before Your Memory's a Waterfall!](https://www.wearedevelopers.com/videos/100073-go-with-the-flow-stop-the-leaks-before-your-memory-s-a-waterfall) - [Efficient deployment and inference of GPU-accelerated LLMs​](https://www.wearedevelopers.com/videos/929-efficient-deployment-and-inference-of-gpu-accelerated-llms) - [Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps](https://www.wearedevelopers.com/videos/1894-shipping-faster-with-less-render-on-cloud-hosting-ai-workloads-and-the-future-of-devops) - [Instant KAI Sandboxes with vCluster: Multi-Tenant, Multi-Scheduler GPU Sharing](https://www.wearedevelopers.com/videos/100333-instant-kai-sandboxes-with-vcluster-multi-tenant-multi-scheduler-gpu-sharing) ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)