Software Engineer (Gen AI Solutions)
Robotics Technologies LLC
Austin, TX, United States
1 day ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.indeed.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Experience required
2 years minimum
Compensation
$158,080
Working hours
Regular working hours
Job source
Tech stack
Application Programming Interfaces (APIs)
C++ (Programming Language)
Profiling
Nvidia CUDA
Continuous Integration
Distributed Systems
Github
Python (Programming Language)
Open Source Technology
Inference Optimization
OpenAI
Prometheus
+33 more
Runbook
Software Engineering
Data Streaming
Datadog
Data Logging
Graphics Processing Unit (GPU)
Pytorch
Retrieval-Augmented Generation
Large Language Models
Grafana
Llamaindex
Generative AI
Rate Limiting
Discretization
Pgvector
Kubernetes
Low Latency
ONNX (Open Neural Network Exchange) Format
HuggingFace
SGLang
Milvus
Triton Inference Server
Machine Learning Operations
FAISS
TensorRT
Evaluation of Large Language Models
Api Design
Google Gemini
Model Inference
Terraform
Splunk
Docker
Jenkins
Job description
We need an experienced software engineer to help design, build, and optimize production LLM inference services that power GenAI features for internal and customer-facing applications. The engineer will work with our ML, platform, and product teams to make model serving faster, cheaper, and more reliable at scale.
Scope of work :
- Design, deploy, and maintain scalable inference services for open-weight and hosted LLMs on Kubernetes-based infrastructure.
- Profile and reduce end-to-end latency (time to first token, tokens per second) and cost per request.
- Apply model and serving optimizations, then benchmark the trade-offs between quality, latency, and throughput.
- Build and maintain integrations with external model APIs and internal orchestration frameworks, including routing, fallback, retries, and rate-limit handling.
- Add observability across the request path, covering logging, metrics, tracing, and dashboards for request volume, downstream call ratios, errors, and token usage.
- Set up evaluation and regression pipelines so model or serving changes donβt degrade output quality.
- Do capacity planning and autoscaling, and track GPU utilization and cost.
- Write technical design docs and runbooks, and hand knowledge over to the internal team.
Expected deliverables :
- Production-ready inference service(s) with CI/CD and infrastructure as code
- Latency and cost benchmark reports, measured before and after optimization
- Monitoring dashboards and alerting
- Architecture docs, runbooks, and handover sessions
Requirements
- Software engineering experience, including 2+ years on ML/LLM systems in production
- Strong Python, plus working knowledge of Go, C++, or Rust for performance-critical paths
- Hands-on experience with LLM serving frameworks: vLLM, TensorRT-LLM, TGI, SGLang, or Triton Inference Server
- Inference optimization techniques:
- Quantization (INT8/INT4, FP8, AWQ, GPTQ)
- KV-cache management, paged attention, prefix caching
- Continuous/dynamic batching and speculative decoding
- Tensor and pipeline parallelism
- Experience with PyTorch and the Hugging Face ecosystem
- Kubernetes, Docker, and GPU scheduling (NVIDIA GPU operator, node pools, autoscaling)
- Integration with hosted LLM APIs (OpenAI, Gemini, Anthropic), including streaming, retries, and rate limiting
- Observability tooling: Prometheus/Grafana, Splunk, Datadog, OpenTelemetry
- Solid grounding in distributed systems, API design (REST/gRPC), and performance profiling
Preferred :
- GenAI orchestration frameworks: Genkit, LangChain, LlamaIndex
- CUDA, Triton kernels, FlashAttention, or other low-level GPU optimization
- Model compilation and runtimes: ONNX Runtime, TorchScript/torch.compile, Core ML
- RAG pipelines and vector databases (FAISS, Milvus, pgvector)
- LLM evaluation frameworks and building automated quality and regression benchmarks
- Cost modeling and FinOps for GPU workloads
- Infrastructure as code (Terraform, Helm) and CI/CD (GitHub Actions, Jenkins)
- Contributions to open-source inference projects, or published work in ML systems
Soft skills :
- Can turn ambiguous requirements into concrete technical plans and trade-off proposals
- Communicates clearly in writing (design docs, benchmark reports)
- Works well with cross-functional teams of engineers and PMs.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.indeed.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦