> Markdown version of [/jobs/ext/3848137-software-engineer](https://www.wearedevelopers.com/jobs/ext/3848137-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Software Engineer

- **Company:** Splunk, Inc.
- **Location:** London, UK
- **Contract:** Permanent contract
- **Skills:** Java (Programming Language), Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Microsoft Azure, C++ (Programming Language), Software as a Service, Cloud Computing, Code Review, Decision Support Systems, Elasticsearch, Github, Python (Programming Language), PostgreSQL, Redis, Site Reliability Engineering Practices, Prometheus, Software Engineering, Software Systems, Data Streaming, Google Cloud, Evaluation Pipelines, Retrieval-Augmented Generation, Large Language Models, Grafana, Mttr, Agentic-AI, Kubernetes, Information Technology, Apache Kafka, Data Management, Model Context Protocol, Terraform, Splunk, Dynatrace, Human in the Loop, Docker, Jenkins, Golang, Microservices
- **Published:** October 21, 2026
- **Apply:** https://www.totaljobs.com/job/software-developer/splunk-job108083836

## About the Role

* Bachelor's degree + 8 years of related experience, Master's + 6 years, or PhD + 3 years in Computer Science, Software Engineering, or a related technical field.
 * Experience as a Senior / Lead SRE or Software Engineer delivering distributed, high-availability SaaS platforms at scale.
 * Strong proficiency in Python, Go, Java, or C++ with experience designing microservices, APIs, and production automation.
 * Deep experience with Kubernetes, Docker, and container orchestration in large-scale multi-cluster environments.
 * Proven background in SRE practices: SLI/SLO design, observability platforms (metrics/logs/traces), incident management, and automated root cause analysis (RCA)., * AI & Agentic Systems: Experience building LLM pipelines, AI Agents, Model Context Protocol (MCP) servers/clients, RAG architectures, and evaluation frameworks.
 * Observability & Telemetry: Experience with OpenTelemetry (OTel), Prometheus, Grafana, Splunk, ThousandEyes, or distributed tracing systems.
 * Cloud & Infrastructure: Expertise in public cloud providers (AWS, GCP, Azure), Terraform/IaC, and GitOps/CI/CD pipelines (Jenkins, GitHub Actions, Harness).
 * Safe Automation & Guardrails: Experience implementing responsible AI guardrails, deterministic fallback logic, and policy-driven remediation engines.
 * Data & Messaging: Experience with streaming and data platforms (Kafka, Redis, PostgreSQL, Elasticsearch/Vector DBs).
 * CollabHiring

## Description

As a Software Engineer, you will design, build, and deploy the core capabilities of our next-generation AI-powered Production Intelligence platform. You will combine Site Reliability Engineering practices with modern agentic AI (reusable Skills, Model Context Protocol (MCP), and LLM tooling) to transform how engineering and leadership teams monitor, diagnose, and auto-remediate global SaaS infrastructure., * Technical Design & Architecture: Design and implement high-resilience software systems for AI-assisted observability, automated incident response, and self-healing cloud infrastructure.

 * Agentic Workflows & Tooling: Design, build, and maintain production-grade AI agents, MCP tool integrations, and deterministic evaluation pipelines for automated operational decision support.
 * Telemetry & Insights: Implement ingestion and correlation pipelines across distributed logs, metrics, OpenTelemetry traces, change events, and runbooks to accelerate Mean Time to Detection (MTTD) and Resolution (MTTR).
 * Safe Production Automation: Develop proactive anomaly detection and Human-in-the-Loop (HITL) remediation workflows with rigorous safety, security, and quality guardrails.
 * Reliability & Scalability Engineering: Partner with application and infrastructure teams to define SLIs/SLOs, handle error budgets, and lead deep-dive post-incident reviews (PIRs).
 * Mentorship & Best Practices: Mentor mid-level and junior engineers, conduct thorough code reviews, establish engineering guidelines, and drive operational excellence across global development and operations teams.
 * Cross-Team Delivery: Manage priorities and deadlines, communicate progress clearly, and work across teams to turn production needs into reliable software and AI-assisted capabilities.