Senior Storage Engineer

We’re hiring a Senior Storage Engineer to design, deploy, operate and troubleshoot the storage that supports Cotidal’s AI infrastructure. You will focus on performance, availability, scalability and data protection, with immediate work that includes day-to-day VAST Data operations., * Design, deploy, operate and maintain large-scale storage for AI/HPC and GPU clusters. Support parallel file systems, scale-out NAS and distributed storage architectures.

• Manage and troubleshoot VAST Data, WEKA and other distributed storage platforms. Optimize throughput, latency, IOPS, checkpointing and data ingestion for AI workloads.
• Configure and troubleshoot NFS, SMB, NVMe-oF and S3-compatible storage access. Support storage networking over high-speed Ethernet and RDMA/RoCEv2.
• Manage capacity, replication, snapshots, data protection and disaster recovery so storage can meet performance and availability needs as deployments grow.
• Investigate bottlenecks across compute, networking and storage, lead incident root-cause analysis, and work with AI infrastructure, compute, networking and platform teams to resolve issues.
• Perform upgrades, firmware maintenance and capacity expansions. Build and maintain the monitoring and automation needed to operate the storage infrastructure.

Storage Software, Supercomputing, Artificial Intelligence (AI), Training/Teaching, Research Laboratory, Hardware Design, Storage Software, Identify Issues, Information/Data Security (InfoSec), Capacity Management, * 10-15 years of professional storage engineering and infrastructure experience, including designing, deploying, operating and troubleshooting enterprise and high-performance storage.

• Strong production experience with enterprise and distributed storage platforms, and hands-on expertise with VAST Data, WEKA or comparable high-performance storage.
• Familiarity with Dell PowerScale/Isilon, NetApp ONTAP, Pure Storage or IBM Storage Scale (GPFS).
• Experience with GPU cluster workloads, parallel I/O, dataset ingestion, checkpointing, performance tuning and bottleneck analysis.
• Knowledge of NFS, SMB, NVMe/TCP, NVMe/RDMA, S3-compatible object storage, RDMA/RoCEv2, TCP/IP and 100G/200G/400G Ethernet.
• Linux administration and experience with Python, Bash, Ansible or Terraform.
• Experience with telemetry, Prometheus/Grafana, incident response, change management and capacity planning.
• The ability to troubleshoot critical performance and availability issues independently, with strong communication, documentation and cross-functional collaboration skills., * Deep VAST Data and WEKA architecture, deployment and performance tuning.
• Lustre, BeeGFS, IBM Storage Scale (GPFS) or Ceph.
• Integration with NVIDIA DGX/HGX or AMD GPU clusters, and NVIDIA GPUDirect Storage (GDS).
• Kubernetes storage and CSI drivers, multi-site replication and data mobility.
• Hyperscale or cloud AI infrastructure and petabyte- to exabyte-scale datasets.

Location: Palo Alto, in person.

Compensation: Competitive salary and equity.

Visa sponsorship: We sponsor work visas and help you navigate the process.

Benefits: Medical, dental, and vision coverage, unlimited PTO, and relocation support as needed.

Supercomputing built for AI.

Cotidal is building the computing infrastructure that AI companies and research labs need to train and run their models. Starting with TPUs, we design the hardware and software together around their workloads, with the goal of making powerful AI chips easier to use at scale.

Our mission is endless compute for humanity. Over time, we want customers to choose across chip stacks without rebuilding their infrastructure.