Splunk Engineer

The IA&E Engineer provides advanced cybersecurity engineering and infrastructure expertise in support of Treasury’s enterprise-wide SOC operations under TESIEMS. This position is responsible for designing, integrating, and maintaining secure compute, storage, and virtualization environments across both on-premises data centers and cloud environments (AWS). Additionally, this role configures and troubleshoots Splunk platform components, focusing on resolving connectivity and performance issues across the Splunk environment. Ensure platform availability, data integrity, and reliability of log flow from multiple sources. The engineer applies deep technical knowledge of Linux systems, virtual machines, and enterprise infrastructure to optimize SOC monitoring, strengthen Treasury’s cybersecurity posture, and ensure resilient, scalable operations across the Department., * Primary responsibilities include design, configure, and maintain low-level Splunk platform components including forwarders, indexers, search heads, and deployment servers.

• Troubleshoot system outages, downtimes, and performance degradation.
• Diagnose and resolve issues with unresponsive or misconfigured hosts and forwarders. Investigate and remediate syslog ingestion issues, including network connectivity, parsing, and logging issues.
• Support integration and troubleshooting of security and email gateway logs, including IronPort data sources.
• Configure, maintain, and optimize Splunk configuration files including props.conf, inputs.conf, transforms.conf, and related configurations.
• Analyze logs to identify root causes of ingestion delays, dropped events, or parsing errors.
• Monitor data pipelines to ensure consistent log ingestion, normalization, and indexing.
• Collaborate with system administrators, network teams, and security teams to resolve cross-platform issues.
• Document troubleshooting procedures, configuration changes, and operational best practices.
• Develop dashboards and Splunk Health Checks to check intermediate forwarder connectivity, search head availability, disk utilization, and license usage.
• Configure and manage server classes to control forwarder configurations, app distribution, and deployment across the environment.
• Manage and troubleshoot Cluster Manager controls, including index replication, bucket fixing, peer status, and cluster health.
• Configure, deploy, and support intermediate forwarders, including load balancing, filtering, routing, and data normalization.
• Coordinate planned maintenance, upgrades, and configuration changes ensuring minimal disruption to data ingestion and search availability.
• Develop and maintain operational runbooks, health check procedures, and escalation workflows.

• Seven (7) years of experience. Three additional years of experience in lieu of degree.
• Deep Splunk technical knowledge and proficiency.
• Problem-solving complex Splunk data flow and configuration technical issues.
• Proven experience supporting Splunk in a production environment with a focus on low-level operations and troubleshooting.
• Strong understanding of Splunk data ingestion pipelines, forwarders, indexing processes, and search performance.
• Hands-on experience configuring and troubleshooting props.conf, inputs.conf, and transforms.conf.
• Strong Linux/Unix system administration skills, including log analysis, process monitoring, and network troubleshooting.
• Excellent problem-solving skills and strong communication skills for coordinating with technical teams.
• Strong analytical and critical thinking skills.
• Understanding of impact assessment on end-products or solutions.
• Broad technical understanding of related cybersecurity specialty areas.
• Ability to develop and implement technical solutions independently.
• Familiarity with incident detection, response, and security event management. Proficiency in tools such as SIEMs (e.g., Splunk), IDS/IPS, endpoint detection, and scripting languages.
• Familiarity with NIST SP 800-53, FISMA, and risk management frameworks.
• Experience with scripting (e.g., Python, Bash) and log data analysis.

DESIRED QUALIFICATIONS:

• Bachelor’s degree from an accredited institute in an area applicable to the position in Cybersecurity, Computer Science, Information Systems, or a related discipline.
• Relevant certifications, such as Certified Information Systems Security Professional (CISSP), Certified Information Security Manager (CISM), Cyber Security Analyst + (CySA+) are highly desirable.

Pueo is known for bringing the best talent and unique tools to every opportunity. Pueo’s Parliament (aka workforce) is composed of professionals who are seeking the opportunity to work in a business organization that thrives on career development and independence. In support of mission and professional growth, our Parliament has supported the development of multiple patents, proprietary tools, and applications as well as trademarked processes.

Our organization emphasizes career development across multiple career environments (at the members own pace) and ensures those who contribute broadly are properly rewarded. Pueo has four career environments where every member of the parliament can participate. Each environment has opportunities available for all levels. Opportunities are framed by an employee’s desires and capabilities, and we ensure challenges, growth, and unique experiences are available for employees at all levels.

Our Career Environments (Program, Functional, Service, and Leadership) provide numerous opportunities for employees to invest in their personal growth and those things that offer fulfillment. We invest in helping our members create and execute their career development plans. Our Pods (small teams of 5 or less) are comprised of personnel with similar skillsets to ensure mentorship, understanding, and peer support.