Site Reliability Engineer

STN, inc.
United States
1 day ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours

Tech stack

Artificial Intelligence Build Automation Ubuntu (Operating System) Command-Line Interface Cloud Computing Computer Clusters Configuration Management Computer Programming Linux DevOps Distributed Systems Python (Programming Language)
+18 more
Network Troubleshooting Network Service Reliability Engineering Software Tools Ansible Prometheus Runbook Server Administration Software Engineering Datadog Graphics Processing Unit (GPU) Grafana Kubernetes Information Technology Deployment Automation Slurm Claude Terraform

Job description

The Site Reliability Engineer keeps GPU One (GPUaaS) reliable, observable, and automated at scale. This role blends deep systems troubleshooting with strong software automation to eliminate toil and ensure the GPU compute platforms that run customer AI workloads stay fast and available. It is a hands-on position: you will work with root access on production head and compute nodes, and you should be as comfortable at an Ubuntu command line as you are in an editor., * Own the reliability, availability, and performance of production Linux GPU clusters, from the operating system up through drivers, GPUs, high-speed networking, and storage

  • Lead deep, end-to-end troubleshooting of complex distributed systems, GPU nodes, networking, and storage issues
  • Troubleshoot and resolve GPU rail and NCCL performance issues across multi-GPU and multi-node collective communication paths
  • Diagnose network faults end to end - configuration, routing, and the physical layer, including cabling, transceivers, and link errors
  • Configure and maintain the workload managers that schedule customer jobs - Kubernetes, Slurm, or both - along with the identity, storage, and networking services they depend on
  • Build automation and tooling that eliminates operational toil, with enough fluency in a modern language such as Python or Go to design the solution, review what is written, and redirect an approach that is heading the wrong way
  • Use AI-assisted engineering tools such as Claude to accelerate automation, runbooks, and incident analysis
  • Automate provisioning, image deployment, configuration, and remediation with Ansible and infrastructure-as-code
  • Design and operate observability with Grafana, Prometheus, and Loki - tuning alerts toward signal and building self-healing that takes a human out of the loop
  • Lead incident response, on-call, blameless postmortems, and the reliability improvements that keep us inside customer SLAs

Requirements

Reliability Engineering, Reliability Engineering, Software Engineering, Cloud Computing, Ubuntu, GPU (Graphics Processing Unit), Artificial Intelligence (AI), Command Line, Computer Systems, Software Administration, * 5+ years in SRE, DevOps, or production engineering roles

  • Strong programming skills in Go, Python, or both
  • Hands-on experience operating Kubernetes-based platforms at scale
  • Deep familiarity with observability tooling (Prometheus, Grafana, Datadog, OpenTelemetry)
  • Strong incident management experience including major-incident command, * 5+ years in systems, infrastructure, or SRE engineering operating production systems at scale
  • Deep Linux troubleshooting skills across OS, networking, storage, and performance, with real comfort working as root on production systems (we run Ubuntu almost exclusively)
  • Hands-on experience operating GPU servers in production - driver, device, and hardware-level problems, not only the workloads running on top of them
  • Practical network troubleshooting, including physical-layer faultsStrong automation instincts and the programming ability behind them, in Python or a comparable language
  • Experience with configuration management, node provisioning, and IaC using Ansible, Terraform, or similar
  • Experience building observability and alerting with Grafana and Prometheus
  • Bachelor’s degree in computer science or equivalent experience

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…