Observability Engineer (Splunk)

SS&C Technologies, Inc.
Boston, United States of America
1 month ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Boston, United States of America

Tech stack

Amazon Web Services (AWS)
Amazon Web Services (AWS)
Backup Devices
Bash
Command-Line Interface
Cloud Computing
Configuration Management
Linux
File Systems
DNS
CURL
Logical Volume Manager
Openshift
Performance Tuning
Regular Expressions
Ansible
Indexer
Amazon Web Services (AWS)
GIT
Kubernetes
Information Technology
Splunk
Data Pipelines

Job description

  • Assist in the day-to-day health, reliability, and performance of Splunk Enterprise (onprem), including installs, upgrades, monitoring, backups, and recovery.
  • Design, operate, and evolve distributed Splunk architectures, including Search Head Clusters (SHC) and Indexer Clusters.
  • Manage ingestion and indexing pipelines end-to-end, including forwarders, parsing rules, and index design/retention.
  • Develop and maintain SPL searches, dashboards, alerts, and data models that are operationally meaningful and actionable.
  • Operate storage and retention strategies; contribute to SmartStore planning/implementation where appropriate.

Observability Enablement

  • Partner with infrastructure and application teams to onboard telemetry correctly and sustainably (logs, metrics, traces, and events).
  • Define and enforce observability standards (naming, tagging, retention, alert thresholds, and dashboard patterns).
  • Improve signal-to-noise: reduce alert fatigue and increase actionable, well-contextualized alerts.

Linux & Infrastructure Operations

  • Execute production changes independently and safely: instance resizing/replacement, storage migrations, network/IP cutovers, and maintenance-window execution.
  • Apply sound engineering judgment to minimize downtime and preserve rollback options.
  • Create and improve runbooks, automation, and post-change verification checklists.

Requirements

Splunk (OnPrem Splunk Enterprise)

  • Hands-on experience administering Splunk Enterprise in onprem environments (not exclusively Splunk Cloud).
  • Strong understanding of configuration layering, knowledge objects, and Splunk's data pipeline (input parsing indexing search).
  • Experience with distributed Splunk (SHC and indexer clustering), including troubleshooting and performance tuning.
  • Working knowledge of SPL, dashboards, and alerting suitable for production operations.

Linux / CLI Fluency (NonNegotiable)

  • Comfortable operating primarily from the command line; can inspect and manipulate files and system state quickly and safely.
  • Bash, systemd, process troubleshooting, and /proc familiarity.
  • Text and data tooling: vi (or equivalent), regex, sed/awk, jq, pipelines.
  • SSH identities & certificates, rsync, RPM packages/repos, tar/zip, DNS basics, curl.
  • Filesystem and storage competence: mounts, disk space management, LVM concepts, growth/resize patterns.
  • Practical Git skills (beyond the happy path)., * Degree in Computer Science or related degree
  • 5+ years of experience
  • Splunk Enterprise Certified Admin (or equivalent experience).
  • SmartStore configuration and operational experience.
  • AWS fundamentals (EC2, EBS, VPC) and hybrid connectivity patterns.
  • Telemetry pipelines from containerized applications; OpenTelemetry familiarity.
  • OpenShift/Kubernetes exposure; configuration management tools (Salt/Ansible); containers/K3S.
  • Ability and willingness to teach and uplift team practices.

What We Value

  • Curiosity and continuous learning.
  • Strong judgment and clear thinking under pressure.
  • Ownership mentality and bias toward automation and repeatability.
  • Communication that is direct, respectful, and documented.

This Role May Not Be a Fit If

  • You require detailed runbooks for every task and avoid ambiguity.
  • You are uncomfortable in a terminal-first workflow.
  • You prioritize speed over safety in production environments.

Benefits & conditions

  • Flexibility: Hybrid Work Model and Business Casual Dress Code, including jeans
  • Your Future: 401k Matching Program, Professional Development Reimbursement
  • Work/Life Balance: Flexible Personal/Vacation Time Off, Sick Leave, Paid Holidays
  • Your Wellbeing: Medical, Dental, Vision, Employee Assistance Program, Parental Leave
  • Wide Ranging Perspectives: Committed to Celebrating the Variety of Backgrounds, Talents and Experiences of Our Employees
  • Training: Hands-On, Team-Customized, including SS&C University
  • Extra Perks: Discounts on fitness clubs, travel and more!

About the company

As a leading financial services and healthcare technology company based on revenue, SS&C is headquartered in Windsor, Connecticut, and has 27,000+ employees in 35 countries. Some 20,000 financial services and healthcare organizations, from the world's largest companies to small and mid-market firms, rely on SS&C for expertise, scale, and technology., Unless explicitly requested or approached by SS&C Technologies, Inc. or any of its affiliated companies, the company will not accept unsolicited resumes from headhunters, recruitment agencies, or fee-based recruitment services. SS&C offers excellent benefits including health, dental, 401k plan, tuition and professional development reimbursement plan.

Apply for this position