> Markdown version of [/jobs/ext/4131716-software-engineer-inference-platform](https://www.wearedevelopers.com/jobs/ext/4131716-software-engineer-inference-platform). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer, Inference Platform

- **Company:** Fluidstack Ltd
- **Location:** San Francisco, CA, United States
- **Experience:** Expert
- **Salary:** $165,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Nvidia CUDA, Computer Programming, Distributed Systems, InfiniBand, Python (Programming Language), Node.Js, Open Source Technology, Performance Tuning, Software Deployment, Software Engineering, Computer Network Operations, Pytorch, Autoscaling, Large Language Models, AWS Trainium, Kubernetes, SGLang, TensorRT, VLLM
- **Published:** October 26, 2026
- **Apply:** https://www.careerbuilder.com/job-details/software-engineer-inference-platform-ca--45dc6648-677c-44cb-84d0-12da11e1c27a

## About the Role

Software Engineering, Network Operations Center, High Tech Industry, Artificial Intelligence (AI), Distributed Computing, Integrated Circuits (ICs), Process Improvement, Incident Response, Performance Tuning/Optimization, Service Level Agreement (SLA), * 5+ years of professional software engineering experience with a track record of shipping production-quality systems.

 * Strong programming skills in Python and/or Go.
 * Hands-on production experience with at least one LLM serving framework (vLLM, SGLang, TensorRT-LLM, TGI, or equivalent).
 * Working knowledge of PyTorch or JAX and an understanding of how model architecture choices affect inference characteristics.
 * Experience deploying and operating GPU workloads on Kubernetes at production scale, including autoscaling and resource scheduling.
 * Solid understanding of GPU memory hierarchies, compute parallelism, and the tradeoffs across tensor, pipeline, and expert parallelism strategies.
 * Ability to create structure from ambiguity and communicate technical tradeoffs clearly to both engineering peers and customers.
 * Great written and verbal communication skills in English., * Production experience with disaggregated prefill/decode architectures (NVIDIA Dynamo, LLM-d, or equivalent), including scheduling policies and network fabric configuration.
 * Deep familiarity with KV cache strategies: RadixAttention, slab-based memory allocators, cross-request prefix sharing, and cache-aware scheduling.
 * Experience with multi-node GPU inference across InfiniBand or RoCE fabrics, including NCCL collective communication tuning.
 * Custom kernel or operator development experience (e.g., CUDA, Triton, torch.compile, Pallas, or equivalent)
 * Contributions to open-source inference engines (vLLM, SGLang, TGI, TensorRT-LLM, or similar).
 * Hands-on experience with quantization tooling: GPTQ, AWQ, FP8 via llm-compressor, or AutoGPTQ.
 * Knowledge of speculative decoding implementations (Medusa, EAGLE-3, draft-model approaches) and their performance/quality tradeoffs.
 * Experience optimizing and adapting model implementations for non-NVIDIA accelerators and their ecosystems: AMD, TPU, Trainium/Inferentia, Cerebras, Groq, and other custom ASICs.

## Description

Inference is now the defining cost and latency bottleneck for frontier AI. Fluidstack's Inference Platform team owns the serving layer that sits between our global accelerator supply and the production workloads our customers run on it: LLM serving frameworks, KV cache infrastructure, disaggregated prefill/decode pipelines, and Kubernetes-based orchestration across multi-datacenter footprints.

This is a hands-on IC role at the intersection of distributed systems, model optimization, and serving infrastructure. You'll own end-to-end inference deployments for frontier AI labs and our inference product, drive measurable improvements in throughput, cost-per-token, and time-to-first-token, and contribute to the platform architecture choices that determine how Fluidstack deploys across tens of thousands of accelerators.

You will:

 * Own inference deployments end-to-end: from initial configuration and performance tuning to production SLA maintenance and incident response.
 * Drive measurable improvements in throughput, TTFT, and cost-per-token across diverse model families (dense transformers, mixture-of-experts, multi-modal) and customer workload patterns.
 * Build and operate KV cache and scheduling infrastructure to maximize utilization across concurrent requests.
 * Implement and validate disaggregated prefill/decode pipelines and the Kubernetes orchestration that supports them at scale.
 * Profile and resolve bottlenecks at the compute, memory, and communication layers; instrument deployments for end-to-end observability.
 * Partner with customers to translate their model architectures, access patterns, and latency requirements into deployment configurations and upstream platform improvements.
 * Contribute to inference platform architecture and roadmap, with a focus on reducing deployment complexity, improving hardware utilization, and expanding support for new model classes and accelerators.
 * Participate in an on-call rotation (up to one week per month) to maintain the reliability and SLA commitments of production deployments.