Founding Machine Learning Infrastructure Engineer

You will work on model serving performance, accelerator utilization, long-context inference, batching, scheduling, KV cache management, runtime efficiency, and cost reduction. This is a deeply technical role at the intersection of ML systems, infrastructure, and product.

Direct TPU experience is a strong plus, but not required. We care most about strong ML systems fundamentals, performance intuition, and the ability to ship reliable systems quickly.

What You’ll Do

• Optimize large-scale LLM inference and serving systems.
• Improve total tokens per second, decode tokens per second, latency, throughput, and cost efficiency.
• Work on serving infrastructure for open-source models across different types of accelerators.
• Improve batching, scheduling, KV cache management, memory usage, and accelerator utilization.
• Support long-context inference, including workloads targeting up to 1M context.
• Debug performance bottlenecks across model execution, runtime, networking, and infrastructure.
• Work with frameworks such as JAX/XLA, PyTorch, vLLM, SGLang, TensorRT-LLM, or related systems.
• Collaborate closely with the application team to ensure infrastructure is optimized for agentic workloads, not just generic chatbot inference.
• Help turn research prototypes into reliable, high-performance production systems.

• Strong experience in ML systems, distributed systems, or high-performance computing.
• Experience optimizing inference or training workloads for large models.
• Familiarity with TPUs, GPUs, or other accelerators.
• Experience with one or more of CUDA, Triton, NCCL, JAX/XLA, PyTorch internals, vLLM, SGLang, TensorRT-LLM, distributed inference, or distributed training.
• Strong systems debugging skills.
• Comfort working across model code, runtime, infrastructure, and product requirements.
• High ownership and the ability to operate effectively in an early-stage startup environment.

Cultural Fit

• Hands-on technical excellence and strong engineering judgment.
• End-to-end ownership, from design to implementation to production outcomes.
• Bias for action: ship quickly, learn from failures, and iterate.
• High intensity during critical milestones, with a focus on real customer impact.
• Ability to do deep, focused work and sustain execution.
• Clear communication with teammates, customers, and stakeholders.
• Comfort with ambiguity, rapid change, and wearing multiple hats.
• Low ego, high integrity, high accountability, and strong collaboration.
• Continuous learning and a belief that judgment, intelligence, and capability compound over time.

If you are excited to build the infrastructure and agent systems behind the next generation of AI applications, push open-source models to production-grade performance, and turn ambitious research ideas into real-world impact, Peano AI is the place for you.