Senior Software Engineer - Distributed Systems

Serving inference at scale is a scheduling problem. Requests arrive with wildly different shapes and latency expectations, GPUs are heterogeneous and expensive, and the difference between a platform that’s fast and one that’s economical usually comes down to how well work gets placed. That system is what you’ll own.

You’ll design and build the scheduling and routing layer of our platform: how requests get admitted, prioritized, batched, and placed across a heterogeneous multi-cloud GPU fleet, under real multi-tenant load and real latency commitments. This is core-systems work with a clean slate - you’ll be making the foundational architectural decisions, not maintaining someone else’s, and the quality of those decisions will show up directly in our margins and our customers’ latency numbers.

You’ll work close to the metal and close to the math. Some days that means reasoning about queueing behavior and control loops on a whiteboard; other days it means profiling Go or Rust until the tail latency comes down. We’re a small team, so you’ll own systems end-to-end - design, implementation, rollout, and the production reality afterward.

Responsibilities

• Own the scheduling and routing layer - design and build request admission, prioritization, batching, and placement across a heterogeneous GPU fleet spanning multiple clouds and accelerator types
• Engineer for latency and utilization at once - drive down tail latency while driving up fleet utilization; these fight each other, and resolving that tension well is the job
• Model the system, not just code it - apply queueing theory, control theory, and load-shedding principles to make the platform behave predictably under bursty, multi-tenant traffic
• Build multi-tenant fairness and isolation - ensure priority guarantees and SLO commitments hold when the fleet is saturated and customers are competing for the same capacity
• Own it in production - instrument, observe, and debug distributed behavior in a live system; carry your designs through rollout and real-world load
• Set the technical bar - make foundational architecture decisions, write the design docs that anchor them, and raise the engineering standard of everyone around you

This is a senior individual-contributor role. We’re looking for someone who has built systems like this before and can operate independently from the first week.

• Substantial experience building and operating large-scale distributed systems in production - you’ve owned something load-bearing, not just contributed to it
• Track record of designing core systems from zero to one, and living with the consequences of your architectural decisions
• Hands-on experience with scheduling, load balancing, request routing, or resource allocation systems
• Strong systems fundamentals - operating systems, networking, concurrency - and the ability to reason quantitatively about system behavior using queueing theory, control theory, or similar
• Fluency in a performance-sensitive language (Go, Rust, or C++), with the profiling and optimization instincts that come from actually chasing latency in production
• Comfort with GPU infrastructure and LLM inference fundamentals - batching, KV cache behavior, throughput/latency tradeoffs; deep expertise here is a plus, but strong distributed-systems judgment matters more
• Clear technical writing - you can make a hard design decision legible to people who weren’t in your head
• An AI-native way of working - you use AI tools daily and have your own view of how they change how infrastructure gets built

Preferred Qualifications

• Mandarin proficiency is a plus

Nice to have: Kubernetes and multi-cloud operations experience; open-source contributions to inference, serving, or scheduling projects; experience operating GPU clusters at scale.

Referral program, 401(k), Health insurance, Retirement plan, Paid time off, Vision insurance, Health savings account, Dental insurance Full-time Hybrid work in Santa Clara, CA 95054, * Ownership of a core system at the foundation of the platform, with the architectural latitude that comes with building it first

• Meaningful equity - we expect the people who build the foundational systems to own a real piece of what they build
• A small, high-caliber team where the distance between a good idea and it running in production is measured in days, The base salary range for this position is $170,000 - $350,000 per year. The range reflects the position across experience levels; actual base salary will be determined by job-related knowledge, skills, experience, and work location, and may fall anywhere within the stated range.

In addition to base salary, this position is eligible for equity in the company, along with medical, dental, and vision coverage, and other benefits., * 401(k)

• Dental insurance
• Health insurance
• Health savings account
• Paid time off
• Referral program
• Retirement plan
• Vision insurance