Real Time Operations Infrastructure Systems...

The senior system administrator is the technical expert responsible for ensuring that TVA’s UNIX , Windows, and Linux based general use distributed platforms are administered in an effective and efficient fashion.

• Is the Subject Matter Expert (SME) for and provides authoritative advice on the centralized OT operations, automation, monitoring, alerting, reporting, and diagnostic systems and tools.

• Performs or directs analysis and development of centralized OT operations, automation, monitoring, alerting, reporting, and diagnostic systems and tools.

• Analyzes emergent OT infrastructure management systems and operating requirements

• Implements integrated system solutions to optimize the performance and availability of OT systems.

• Develops and implements custom solutions using programming/scripts for efficient operation and to maintain the highest availability of OT systems.

• Installs, customizes, and integrates IT/OT management, monitoring, alerting, and reporting systems (hardware and software) plus ensures the reliability, availability, and performance of those systems.

• Ensures Control Center datacenters are implemented per design specifications and efficient use including rack space, power (primary and backup systems), cooling, and security. Coordinates datacenter monitoring and alert system test to ensure correct and complete system function.

• Establishes infrastructure resource baselines and monitors actual infrastructure resource utilization, volume, and performance on an ongoing basis.

• Develops and customizes Simple Network Management Protocol (SNMP) monitors, traps, and reports and integrates it with other management and monitoring systems.

• Provides training to other OT personnel on the correct and efficient use of implemented systems and reports.

• Establishing guidelines, policies, and procedures governing all system administration activities by junior and intermediate system administrators

• Designing and configuring new server installations; planning and performing hardware and software upgrades to existing server installations; providing cost-benefit analysis for new installations and upgrades

• Functioning as an expert resource to lower level administrators and/or TVA management in one or more of the operating systems used by general usage distributed platforms

• Analyzing server current general usage to distributed environment and recommending upgrades and consolidations when required

• Ensuring the security and integrity of general usage distributed platforms; and maintaining current in industry-wide hardware and software capabilities and practices. Responsible for analysis of server security issues and formulating appropriate solutions involving the use of system security features

• Maintain the infrastructure to assure compliance with configurations, patches and approved designs. Active participant in the change management process to properly execute change with minimum impact to the services provided.

• Develop and implement policies, procedures, and associated training for managing infrastructure, resources, administration, appropriate use, and disaster recovery

• Participate with OT Server Administrator peers as needed to properly support the OT infrastructure on a 24x7 basis including on-call duties, shift work, and emergency support can be required including travel to various locations away from the employee’s primary work site.

• Is the Subject Matter Expert (SME) for NERC CIP Regulatory requirements within TOPS Control Centers

• Other duties as assigned, including but not limited to the participation in audits, security reviews, incident investigations, and presentations for TVA at professional events.

Supervision Received

Independently performs assignments with instructions as to general results expected. Incumbent plans and executes the work with minimal supervision and works under the direction of a department manager.

• Expert knowledge of UNIX, Windows Server, LINUX, Solaris, and VMWare

• Expert knowledge of backup solutions vSphere Data Protection (VDP) Acronis, and Netbackup

• Expert knowledge of Storage Area Network (SAN) that includes SAN Recovery point Appliance and SAN Service Processor

• Expert level of Active Directory and Group Policy, Clustering, software promotion, HP Hardware Support

• Expert level of working knowledge if TVA High Availability systems and Critical Infrastructure

• Expert level knowledge in current and emerging state monitoring, alerting, management and operations of a large enterprise with multiple datacenters.

• Expert level knowledge in plus one or more IT/OT infrastructure technologies such as networking, client server architectures, server and desktop operating systems, databases, server and desktop virtualization, storage area networks, messaging, monitoring systems and tools, dashboard tools, Structured Query Language, scripting languages, etc.

• Knowledgeable in ITIL and Business and Service Management concepts such as UCMDB. General knowledge and understanding of a broad range of management, computer and networking technologies. Ability to analyze and assess computer/network data and logs.

• Ability to plan, develop and coordinate multiple tasks. Ability to communicate both written and oral to clearly communicate business & technical information.

• Proficiency with personal computers and productivity applications such as word processors, spreadsheets, scheduling, presentation, and collaboration applications.

• Ability to work with a cross-functional team of IT/OT professionals.

• Expert level of Server Infrastructure Tools that are for used for patching, automation, hardware monitoring, syslog, server log monitoring, anti-virus and malware protection, configuration management, baseline management.

Minimum Qualifications(Consistent with Classification Standard)

Education and Experience/Certificates, Licenses, and Registration Requirements/Other Requirements

A Bachelor’s degree in computer science, engineering, mathematics or related field of study; or equivalent education, training & experience. Expert knowledge and extensive experience in one or more IT/OT infrastructure technologies such as management/monitoring systems, datacenter operations, networking, client server architectures, server and desktop operating systems, databases, server and desktop virtualization, storage area networks, etc. Travel as required to carry out project work or perform temporary duty at other than home-based site which supports both internal and external business on the schedule and shift requirements of those businesses. Must be able to qualify for and maintain an ADP II and must be able to maintain elevated clearances including but not limited to, NERC-CIP, Sensitive, and / or Non-Nuclear Fitness for Duty clearance. Must be able and willing to assume on-call rotational assignments, which may include 24 x 7 availability