Splunk Engineer
Role details
Job location
Tech stack
Job description
We are seeking a Principal Splunk Engineer to lead the design, operation, and evolution of our large-scale Splunk Enterprise / Splunk Cloud deployment. The platform ingests multi-terabyte daily data volumes across security, infrastructure, and application domains and is a critical component of our SOC and threat-detection capabilities. The ideal candidate has deep expertise in Splunk architecture, large-scale data onboarding, performance optimization, SmartStore/Indexer Clustering, and security-focused use cases., Platform Architecture & Operations:
- Architect, operate, and optimize a distributed, large-scale Splunk environment (indexer clusters, search head clusters, cluster masters, deployment servers, IDM, ADFS/SAML integrations)
- Lead capacity planning, index design, data retention strategies, and SmartStore lifecycle management
- Maintain high availability, scaling, and resilience across multi-site deployments (including DR strategy)
- Drive Splunk version upgrades, app updates, cluster maintenance, and platform hardening
Security Logging & SOC Enablement:
- Collaborate with SOC, Incident Response, and Threat Hunting teams to ensure high-quality security log ingestion
- Onboard and normalize logs from firewalls, EDR, identity platforms, cloud providers, network telemetry, and custom applications
- Develop and optimize detection content: correlation searches, risk-based alerting, data models, macros, lookups, summaries
- Ensure compliance with logging standards (MITRE ATT&CK mapping, CIS/SOC2/ISO27001 logging requirements)
Data Engineering & Observability:
- Build and manage ingestion pipelines, parsing, field extractions, CIM compliance, HEC configurations, and forwarder architecture
- Implement data lifecycle tiers, filtering strategies, routing, and ingestion controls to reduce cost and improve efficiency
- Optimize search performance, knowledge objects, summary indexing, and acceleration strategies
Governance & Best Practices:
- Establish Splunk development standards, dashboards, and naming conventions
- Mentor junior engineers and act as a technical escalation point for the team
- Maintain documentation, operational runbooks, and logging onboarding guidelines
- Partner with Engineering, Cloud, SecOps, and App teams to drive company-wide observability maturity
Requirements
5+ years experience administering large Splunk Enterprise or Splunk Cloud environments
Strong hands-on knowledge of:
- Indexer clustering, search head clustering
- SmartStore / S3-compatible object store design
- Universal/heavy forwarder architecture
- Ingest actions, parsing, props/transforms
- KVStore, RBAC, SAML, encryption
Deep experience with security log ingestion and SIEM use cases
Strong SPL expertise, including:
- Search optimization
- Summary indexing / data model acceleration
- CIM mapping and field normalization
Experience with Linux systems engineering, scripting (Python/Bash), and automation frameworks (Ansible, Terraform, GitOps preferred)
Preferred Qualifications:
Splunk certifications (Core Consultant, Enterprise Admin, Enterprise Architect, ES Analyst/ES Admin, or equivalent)
Experience with:
- Enterprise Security (ES)
- SOAR (Phantom or comparable)
- AWS/Azure/GCP cloud logging architectures
Familiarity with high-throughput message brokers (Kafka/FluentD/Cribl)
Background in cybersecurity engineering or threat detection
Skills:
- Automation
- Influence
- Result Orientation
- Stakeholder Management
- Technical Strategy Development
- Application Development
- Architecture
- Business Acumen
- Risk Management
- Solution Design
- Agile Practices
- Analytical Thinking
- Collaboration
- Data Management
- Solution Delivery Process