> Markdown version of [/jobs/ext/4031572-inference-infrastructure-engineer-serving](https://www.wearedevelopers.com/jobs/ext/4031572-inference-infrastructure-engineer-serving). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Inference Infrastructure Engineer, Serving

- **Company:** Elorian, LLC
- **Location:** Palo Alto, CA, United States
- **Experience:** Experienced
- **Salary:** $200,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, C++ (Programming Language), Nvidia CUDA, Computer Programming, Python (Programming Language), Open Source Technology, Inference Optimization, Azure Machine Learning, System Programming, Scripting, Graphics Processing Unit (GPU), Load Balancing, Autoscaling, Large Language Models, Discretization, Low Latency, SGLang, TensorRT, VLLM, Decoding
- **Published:** October 30, 2026
- **Apply:** https://www.careerbuilder.com/job-details/inference-infrastructure-engineer-serving-palo-alto-ca--a0ebc021-ea4b-4eb8-bd88-a179224bf983

## About the Role

Seed Funding, Artificial Intelligence (AI), GPU (Graphics Processing Unit), Load Balancing, Autoscaling, High Throughput, Python Programming/Scripting Language, Systems/Internals Programming, C++ Programming Language, Computer Programming, * 3+ years of experience building low-latency, high-throughput inference serving systems for large models

 * Strong knowledge of inference optimization techniques (quantization, batching, speculative decoding, KV cache management)
 * Hands-on experience with serving frameworks such as vLLM, TensorRT-LLM, Triton, or SGLang
 * Experience with multi-GPU/multi-node model parallelism for serving (tensor or pipeline parallel)
 * Strong systems programming skills; C++/CUDA a plus alongside Python
 * Experience with autoscaling and load balancing for production ML services
 * A track record of GPU cost optimization at scale

Preferred qualifications (strong candidates may have some, not all):

 * Experience serving multimodal (vision+ language) models
 * Contributions to open-source ML or systems infrastructure projects (e.g., vLLM, SGLang, TensorRT-LLM, Triton)
 * A bias for action and comfort working across stacks and teams in an early-stage environment

## Description

We're looking for an infrastructure engineer to design, optimize, and scale the systems that serve our large multimodal models. Your work will make inference faster, more cost-effective, and more reliable, so our teams can focus on advancing model capabilities rather than managing bottlenecks.

Our focus is on performant, efficient inference, both to power real-world applications and to accelerate research. This role owns the infrastructure that ensures every deployment and evaluation runs smoothly at scale for our visual foundation models.

What You Will Do

 * Build low-latency, high-throughput inference serving systems for our large multimodal models
 * Design and implement techniques that improve latency, throughput, and efficiency, including quantization, batching, speculative decoding, and KV cache management
 * Optimize our codebase and GPU fleet to fully utilize hardware FLOPs, bandwidth, and memory
 * Implement multi-GPU and multi-node model parallelism for serving (tensor or pipeline parallel)
 * Build autoscaling and load balancing for production ML services
 * Establish standards for reliability, observability, and reproducibility across the inference stack
 * Collaborate with researchers to enable high-performance inference for novel architectures