> Markdown version of [/jobs/ext/4200955-staff-site-reliability-engineer-splunk](https://www.wearedevelopers.com/jobs/ext/4200955-staff-site-reliability-engineer-splunk). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Staff Site Reliability Engineer Splunk

- **Company:** Okta, Inc.
- **Location:** New York, NY, United States
- **Experience:** Expert
- **Salary:** $194,000.0
- **Contract:** Permanent contract
- **Skills:** Amazon Web Services, Systems Engineering, Cloud Computing, Computer Programming, Software Debugging, DevOps, Distributed Systems, Domain Name System (DNS), Python (Programming Language), Linux Kernel, Reliability Engineering, Ruby, TCP/IP, Load Balancing, Kubernetes, Low Latency, Data Analytics, Software Coding, Terraform, Splunk
- **Published:** October 31, 2026
- **Apply:** https://www.juju.com/job/1_6197C085E3024ED887877E548E277C27-NY1702606

## About the Role

You will treat infrastructure as code-utilizing Terraform and strong coding proficiency in Go, Python, or Ruby-to automate the deployment of agents and collectors across complex distributed systems.Key Responsibilities * Automated Infrastructure: Design, build, and maintain scalable observability infrastructure using tools like Terraform. * Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services * Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development." * Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.Required Skills & Experience (The Essentials)Log Management: Minimum 5+ Experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization.
Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.SRE Mindset: Minimum 5+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems. * Programming Proficiency: Strong coding skills in SPL, Go for building internal tools and automating workflows. * Distributed Systems: Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS). * Problem Solving: A data-driven approach to debugging complex, cross-service performance bottlenecks.Bonus Skills (The "Nice-to-Haves") * Telemetry Standards: Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications. * Charge-back app: Experience in implementing Splunk charge-back app for usage reportingCloud Platforms: Experience managing observability native tools within AWS or GCP.Additional requirements: * This
position requires the ability to access federal environments and/or have access to protected federal data.
As a condition of employment for this position, the successful candidate must be able to submit documentation establishing U.S. Person status (e.g. a U.S. Citizen, National, Lawful Permanent Resident, Refugee, or Asylee.