> Markdown version of [/jobs/ext/4774722-principal-engineer-software](https://www.wearedevelopers.com/jobs/ext/4774722-principal-engineer-software). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Engineer Software - **Company:** Palo Alto Networks - **Location:** Palo Alto, CA, United States - **Experience:** Expert - **Salary:** $147,000 - $237,500 - **Contract:** Permanent contract - **Skills:** IEEE 802.1Q, Link Aggregation (Ethernet), Artificial Intelligence, Computing Platforms, Intelligent Platform Management Interface, Bash Shell, Border Gateway Protocol, Computer Clusters, Data Centers, Data Infrastructure, Data Center Infrastructure Management (CIM), Linux, Microprocessors, Ethernet, Firmware, InfiniBand, Internet Security, Python (Programming Language), Kernel-Based Virtual Machine, Nginx, OpenShift, Red Hat Enterprise Linux, Reliability Engineering, Software Tools, Ansible, Prometheus, Simple Network Management Protocols, Software Engineering, Virtualization Technology, VMware VSphere, Ceph (Software), Pulumi, Scripting, Graphics Processing Unit (GPU), Load Balancing, Computer Network Operations, System Availability, Grafana, Infrastructure as Code (IaC), Kubernetes, Information Technology, Hardware Asset Management, Coreos, Api Management, Cisco, Nvme - **Published:** November 18, 2026 - **Apply:** https://www.careerbuilder.com/job-details/principal-engineer-software-palo-alto-ca--f983c37e-bb58-4c91-a855-822cca3f74fe ## About the Role Engineering Software, Leading Edge Technology, Internet Security, Artificial Intelligence (AI), Problem Solving Skills, Network Operations Center, Red Hat Linux Operating System, High Availability, Systems Maintenance, Reliability Engineering, * Education: Bachelor's degree in Computer Science, IT, or equivalent experience. * Platform Expertise: 5+ years of experience specifically operating Red Hat OpenShift (OCP) in a production environment. * Hardware Fluency: Deep experience racking/stacking and cabling high-density GPU systems (e.g., NVIDIA DGX or similar) and specialized AI/ML hardware. * Infrastructure as Code (IaC): Advanced proficiency in Ansible or Pulumi for automating bare-metal provisioning and cluster configuration. * Scripting: Strong Python and Bash skills for developing custom health-check scripts and API integrations. * Linux Mastery: Expert-level CoreOS and RHEL administration, including kernel tuning and systemd management. * Networking: Solid understanding of BGP, VLAN tagging, LACP, and Load Balancing (F5/NGINX) essential for cluster ingress. * Virtualization & Storage: Experience with vSphere or KVM, and persistent storage solutions like OpenShift Data Foundation (ODF) or Ceph. * Tooling: Familiarity with DCIM tools (Netbox) and monitoring stacks ( ELK/Lok ..etci). Physical Requirements * Lifting: Ability to lift and move equipment up to 50 pounds (e.g., high-density 2U/4U servers). * Environment: Comfortable working in high-decibel, climate-controlled data center aisles. * Dexterity: Capable of standing, walking, and performing precision cabling in tight rack spaces for extended periods. * Travel: May require occasional travel to remote data center sites or edge locations. ## Description The Senior Data Center Operations Engineer is responsible for the bedrock of our high-availability infrastructure. This role bridges the gap between physical hardware and the Red Hat OpenShift Container Platform (OCP). Your mission is to ensure 99.99% availability by architecting resilient physical layouts and automating the deployment, scaling, and self-healing capabilities of our production clusters., * High-Availability (HA) Infrastructure: Monitor and maintain data center systems with a focus on "Zero Single Point of Failure" (ZSPoF) architecture for OpenShift control planes and worker nodes. * Cluster Reliability Engineering: Implement and manage OpenShift 4.x clusters across multiple power and cooling zones to ensure 99.99% uptime. * Disaster Recovery & Business Continuity: Design, test, and execute automated failover strategies and backup/restore procedures using tools like OADP (Velero) and Red Hat ACM. * Automated Maintenance: Perform routine maintenance and upgrades using GitOps (ArgoCD) and the Machine Config Operator to ensure zero-downtime node evacuations and patching. * Complex Troubleshooting: Resolve deep-stack hardware and software issues, from faulty GPU firmware to OpenShift SDN (OVN-Kubernetes) network latencies. * Vendor & Lifecycle Management: Coordinate with vendors for specialized hardware (e.g., NVIDIA, Dell, Cisco) while maintaining strict security and firmware compliance. * Efficiency & Capacity Architecture: Optimize rack density for high-performance GPU clusters while managing thermal loads and power distribution (PDU) to prevent circuit-trip outages. * Observability Implementation: Maintain accurate documentation and integrate hardware health metrics (IPMI/SNMP) into Prometheus/Grafana for proactive alerting. * Physical Deployment: Rack and stack high-density GPU servers, ensuring redundant power-pathing and high-speed (100G/200G) InfiniBand or Ethernet cabling. * Hardware Lifecycle: Perform precision physical installation and replacement of critical components (CPUs, GPUs, NVMe storage) in a live production environment without impacting cluster quorum.