Software Engineer, ML Infrastructure

Role OverviewWe are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production.This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability.You will design and operate the full ML serving stack from model artifacts to GPU execution, and work closely with Product and ML teams to ensure our models can support high QPS, strict SLAs, and production correctness.This role is ideal for someone who deeply understands how LLMs work internally, but chooses to specialize in making them fast, stable, and production-ready.About Realm LabsRealm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises.Key ResponsibilitiesOwn the end-to-end LLM inference stack, including:Model loading and executionGPU utilization and memory efficiencyRuntime performance tuningProduction deployment and scalingDesign and operate high-performance LLM serving systems using technologies such as:vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLangOptimize inference across:LatencyThroughput (QPS)GPU memory footprintCost efficiencyWork hands-on with PyTorch and TensorFlow models, including:Model graph understandingAttention mechanisms, KV cache behavior, batching strategiesPrecision tradeoffs (FP16, BF16, INT8, etc.)Build and maintain production-grade GPU services:Multi-model servingAutoscaling strategiesFault isolation and graceful degradationCollaborate with application and platform teams to:Define serving APIsEnsure correctness and safety of outputsDebug production issues end-to-endBuild a reproducible model training and versioning system for customer deploymentsEstablish best practices for:Model versioningRollouts and rollbacksPerformance benchmarkingProduction validationExpected Qualifications5+ years of professional experience in ML infrastructure, systems engineering, or production ML roles.Strong software engineering fundamentals; ability to write robust, maintainable production code.Deep hands-on experience with LLM inference infrastructure, including:PyTorch (required)TensorFlow (working knowledge)Proven experience with GPU inference optimization, including:TensorRT / TensorRT-LLMvLLMTriton Inference ServerSGLang or similar serving runtimesStrong understanding of LLM internals, such as:Transformer architecturesAttention and KV cachingBatching, streaming, and token-level generationExperience running ML systems in production with high traffic and SLAsComfortable working in Linux-based, cloud production environmentsPreferred QualificationsExperience deploying LLMs on Kubernetes and GPU clusters.Familiarity with CUDA, NCCL, or low-level GPU performance concepts.Experience with:Model sharding and parallelism strategiesMulti-GPU inferenceStreaming inference systemsKnowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).Experience working at startups or owning systems with minimal abstraction layers.Additional InformationThis is a founding, high-ownership role with direct impact on core product capabilities.You will be expected to build, run, and own systems end-to-end.The role may include limited on-call responsibilities aligned with production ownership.Compensation & BenefitsMarket aligned compensation and benefitsFounding engineer equity (Equity is a significant component of this role and will be discussed)Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc.Visa sponsorship: We do sponsor visas! However, we aren’t able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this. Role OverviewWe are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production.This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability.You will design and operate the full ML serving stack from model artifacts to GPU execution, and work closely with Product and ML teams to ensure our models can support high QPS, strict SLAs, and production correctness.This role is ideal for someone who deeply understands how LLMs work internally, but chooses to specialize in making them fast, stable, and production-ready.About Realm LabsRealm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises.Key ResponsibilitiesOwn the end-to-end LLM inference stack, including:Model loading and executionGPU utilization and memory efficiencyRuntime performance tuningProduction deployment and scalingDesign and operate high-performance LLM serving systems using technologies such as:vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLangOptimize inference across:LatencyThroughput (QPS)GPU memory footprintCost efficiencyWork hands-on with PyTorch and TensorFlow models, including:Model graph understandingAttention mechanisms, KV cache behavior, batching strategiesPrecision tradeoffs (FP16, BF16, INT8, etc.)Build and maintain production-grade GPU services:Multi-model servingAutoscaling strategiesFault isolation and graceful degradationCollaborate with application and platform teams to:Define serving APIsEnsure correctness and safety of outputsDebug production issues end-to-endBuild a reproducible model training and versioning system for customer deploymentsEstablish best practices for:Model versioningRollouts and rollbacksPerformance benchmarkingProduction validationExpected Qualifications5+ years of professional experience in ML infrastructure, systems engineering, or production ML roles.Strong software engineering fundamentals; ability to write robust, maintainable production code.Deep hands-on experience with LLM inference infrastructure, including:PyTorch (required)TensorFlow (working knowledge)Proven experience with GPU inference optimization, including:TensorRT / TensorRT-LLMvLLMTriton Inference ServerSGLang or similar serving runtimesStrong understanding of LLM internals, such as:Transformer architecturesAttention and KV cachingBatching, streaming, and token-level generationExperience running ML systems in production with high traffic and SLAsComfortable working in Linux-based, cloud production environmentsPreferred QualificationsExperience deploying LLMs on Kubernetes and GPU clusters.Familiarity with CUDA, NCCL, or low-level GPU performance concepts.Experience with:Model sharding and parallelism strategiesMulti-GPU inferenceStreaming inference systemsKnowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).Experience working at startups or owning systems with minimal abstraction layers.Additional InformationThis is a founding, high-ownership role with direct impact on core product capabilities.You will be expected to build, run, and own systems end-to-end.The role may include limited on-call responsibilities aligned with production ownership.Compensation & BenefitsMarket aligned compensation and benefitsFounding engineer equity (Equity is a significant component of this role and will be discussed)Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc.Visa sponsorship: We do sponsor visas! However, we aren’t able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this. We are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production. This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability. You will design and operate the full ML serving stack from model artifacts to GPU execution, and work closely with Product and ML teams to ensure our models can support high QPS, strict SLAs, and production correctness. This role is ideal for someone who deeply understands how LLMs work internally, but chooses to specialize in making them fast, stable, and production-ready. Realm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises. Visa sponsorship: We do sponsor visas! However, we aren’t able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this.

• 5+ years of professional experience in ML infrastructure, systems engineering, or production ML roles.
• Strong software engineering fundamentals; ability to write robust, maintainable production code.
• Deep hands-on experience with LLM inference infrastructure, including:
• PyTorch (required)
• TensorFlow (working knowledge)
• Proven experience with GPU inference optimization, including:
• TensorRT / TensorRT-LLM
• vLLM
• Triton Inference Server
• SGLang or similar serving runtimes
• Strong understanding of LLM internals, such as:
• Transformer architectures
• Attention and KV caching
• Batching, streaming, and token-level generation
• Experience running ML systems in production with high traffic and SLAs
• Comfortable working in Linux-based, cloud production environments, * Experience deploying LLMs on Kubernetes and GPU clusters.
• Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
• Experience with:
• Model sharding and parallelism strategies
• Multi-GPU inference
• Streaming inference systems
• Knowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).
• Experience working at startups or owning systems with minimal abstraction layers.

Realm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises.