Senior AI Infrastructure Engineer

You’ll own the technical depth of our inference stack: how models are loaded, served, optimized, observed, and scaled across different GPU hardware.

You’ll work hands-on with systems such as vLLM, SGLang, TensorRT-LLM, CUDA and ROCm, digging below the API layer into what actually determines inference performance.

You’ll:

• Deploy and operate production model-serving infrastructure using vLLM and similar runtimes
• Optimize TTFT, inter-token latency, throughput, GPU utilization, memory usage, and cost
• Tune batching, KV cache behavior, parallelism, quantization, model configuration, and runtime parameters
• Benchmark models across different NVIDIA and AMD GPU architectures
• Diagnose GPU memory, driver, runtime, networking, and performance bottlenecks
• Build reliable patterns for multi-GPU and multi-node inference
• Improve how StackYak provisions, configures, monitors, and operates inference workloads
• Help determine the right hardware and serving configuration for different models and customer workloads
• Build the telemetry and tooling needed to understand inference performance in production
• Work closely with the rest of the engineering team as we turn this operational knowledge into the StackYak platform

We’re particularly interested in engineers who have actually operated LLM inference infrastructure, not just consumed hosted model APIs.

You should have strong experience with several of:

• vLLM, SGLang, TensorRT-LLM, TGI, Triton, or comparable inference runtimes
• NVIDIA GPUs, CUDA, NCCL and the surrounding software stack
• GPU profiling, benchmarking, memory management, and performance optimization
• Quantization and model formats such as FP8, INT8, AWQ or GPTQ
• Tensor parallelism, pipeline parallelism, and distributed inference
• Docker and Linux systems engineering
• Python and the modern ML ecosystem
• Production observability and performance troubleshooting

Experience with AMD GPUs/ROCm, high-performance networking, RDMA/InfiniBand/RoCE, or operating GPU infrastructure in data centers is especially useful.

You do not need to be an ML researcher. We’re looking for someone who understands the systems engineering of inference: what happens between receiving a request and getting tokens out of a GPU, and how to make every part of that path better.

This is a hands-on engineering role at an early-stage company. You’ll have significant influence over how StackYak’s inference infrastructure is designed and how we run models across an increasingly diverse fleet of GPU hardware.

Pay: $160,000.00 - $200,000.00 per year