Model Routing Engineer, zPLATFORM

ZETTABYTES, INC.
Palo Alto, CA, United States
6 days ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
4 years minimum
Compensation
$190,000 - $245,000
Working hours
Regular working hours

Tech stack

A/B Testing Application Programming Interfaces (APIs) Artificial Intelligence Software Applications Cloud Computing Computer Programming Data Centers Distributed Systems Failover Fault Tolerance Python (Programming Language) Routing
+10 more
Network Routers Graphics Processing Unit (GPU) Computer Network Operations System Availability Large Language Models Backend Low Latency Data Analytics VLLM Model Inference

Job description

Model Routing Engineer, zPLATFORMSkip to main content, * AI computing company: we turn GPU hardware into reliable, high-utilisation AI compute through software, orchestration and full lifecycle operations

  • What we offer: enterprise GPU cloud and AI inference, software for data centre owners to run their own GPU clouds, turnkey and sovereign AI data centres, edge computing and GPU financing
  • Who we work with: GPU owners, data centre operators, enterprises, OEMs, integrators and financiers across multiple markets
  • How we work: fast-moving and high-accountability; we value speed, ambition, clarity, ownership and measurable outcomes

Role Summary

The Model Routing Engineer, zPLATFORM builds the services that route each inference request on zPLATFORM to the right model and capacity, balancing quality, latency and cost.

The role works on the request path, the policies that guide routing decisions and the measurement that shows whether those decisions are good.

The Model Routing Engineer partners with inference, platform and product teams to give customers one simple interface across many models.

Reporting Line

Reports to: Co-CTO and CPO, * Routing services: build and operate low-latency, highly available services that direct inference requests across models and regions.

  • Routing policies: implement policies that weigh quality, latency, cost and customer preferences, and make them configurable.
  • Evaluation: build evaluation sets and online metrics that show the effect of routing choices on output quality and cost.
  • Model onboarding: add new open and commercial models to zPLATFORM with consistent APIs and documentation.
  • Reliability: handle failover, rate limits and graceful degradation when models or capacity are unavailable.
  • Regional and compliance rules: apply data residency and customer restrictions in routing, and follow export-control and sanctions screening requirements with the Export Compliance team.
  • Observability: instrument the request path so that latency, errors and cost per request are visible.
  • Customer feedback: work with product and solutions teams to turn customer needs into routing features., * Routing decisions are measured against clear quality, latency and cost metrics
  • New models are added to zPLATFORM through a documented, repeatable process
  • Failover behaviour is tested and documented
  • Customers can set routing preferences through the API

Why This Role Matters

  • Product core: routing between models is a defining capability of zPLATFORM.
  • Customer value: good routing gives customers better answers for less money without extra work on their side.

Working Norms at Zettabyte

  • Bias to action: deliver drafts quickly, then iterate on feedback
  • Be precise: double-check numbers and facts; sloppy analysis creates downstream cost
  • Make it usable: package outputs for decision-making, not data dumps
  • Own the process: whoever drives a deliverable manages stakeholders and timelines
  • Learn the ecosystem: understand GPUs, cluster operations and the buyer landscape

Requirements

Network Routers, Cloud Computing, Artificial Intelligence (AI), GPU (Graphics Processing Unit), High Tech Industry, Computer Software, Network Operations Center, OEM (Original Equipment Manufacturer), Network Routers, High Availability, * 4+ years of backend or distributed systems engineering in production

  • Strong programming skills in Go, Rust or Python
  • Experience building low-latency APIs, gateways or proxies at scale
  • Working knowledge of LLM APIs, inference serving and evaluation methods
  • Comfort with experimentation, A/B testing and data-driven decisions
  • Clear written communication of design trade-offs

Bonus Qualifications

  • Experience with LLM gateways or model routing products
  • Background in recommendation, ranking or search systems
  • Familiarity with inference serving frameworks such as vLLM
  • Experience with multi-region service deployments

Benefits & conditions

  • Compensation: USD 190,000 to 245,000 a year base salary, plus equity
  • Benefits: Medical, dental and vision insurance; 401(k) retirement plan; paid time off and US public holidays; equity
  • Work authorisation: Required in the United States
  • Equal opportunity: Zettabyte is an equal opportunity employer; qualified applicants with arrest or conviction records will be considered in line with the California Fair Chance Act

Model Routing Engineer, zPLATFORM

About the company

Zettabyte is a supplier of the United Nations Global Marketplace (UNGM #1117361) and is committed to aiding the United Nations in achieving its Sustainable Development Goals.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…