HPC Engineer | Experienced Hire

Susquehanna International Group, LLP
Lower Merion Township, PA, United States
2 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Experienced
Working hours
Regular working hours
Job source

Tech stack

C (Programming Language) Artificial Intelligence Bash Shell Big Data C++ (Programming Language) Data Compression Extract Transform Load (ETL) Distributed Computing Environment Distributed Data Store Distributed Systems Ethernet General Parallel File Systems
+14 more
InfiniBand Job Scheduling Python (Programming Language) Linux System Administration Machine Learning Node.Js Performance Tuning Simple Data Format High Performance Computing Feature Engineering Pytorch Slurm Machine Learning Operations Data Pipelines

Job description

We are looking for experienced HPC Engineers to design, build, and optimize the high-performance computing and machine learning infrastructure that powers our research and trading environment. This role spans the full stack: compute, GPU, networking, storage, and scheduling. The goal is to deliver scalable, reliable, high-throughput infrastructure for compute-intensive and data-intensive workloads, including large-scale model training.

You will work closely with researchers, ML engineers, and platform, networking, and data teams to ensure that every layer of the stack, from the interconnect to the filesystem to the scheduler, performs well together.

This posting represents multiple roles across our HPC team, and we don’t expect any one candidate to cover everything below. If you’re strong in some of these areas, we’d like to hear from you.

What you’ll do

  • Design, deploy, and operate HPC and ML infrastructure across CPU and GPU compute clusters, high-speed networks, and parallel storage systems
  • Support and optimize distributed training workloads (e.g., PyTorch), including multi-node, multi-GPU jobs that depend on efficient GPU interconnects and collective communication (e.g., NCCL)
  • Operate and tune high-performance interconnects, including InfiniBand and Ethernet fabrics, to maximize throughput and minimize latency for training and data-intensive jobs
  • Manage parallel and distributed storage systems (e.g., VAST, DDN, Lustre, GPFS, NFS) and ensure data is served efficiently to compute, including decisions about what is served from parallel storage versus held in memory
  • Optimize data pipelines and I/O patterns for large-scale workloads, including file formats, compression strategies, and feature engineering inputs
  • Own data movement workflows across environments, including ingest, replication, tiering, and archiving
  • Administer and improve job scheduling and workload management using Slurm and meta-schedulers (e.g., HyperQueue, Ray) to drive fair, efficient utilization of shared resources
  • Troubleshoot complex performance and reliability issues across the compute, GPU, network, and storage stack
  • Develop and maintain automation for provisioning, monitoring, and lifecycle management of HPC and ML infrastructure
  • Advise researchers and application teams on best practices for job design, data layout, I/O patterns, and performance tuning
  • Evaluate and integrate new hardware, software, and architectures as requirements evolve

Requirements

  • Hands-on experience operating HPC or ML infrastructure in production environments
  • Strong Linux systems administration skills
  • Experience supporting distributed training workloads (e.g., PyTorch) and an understanding of GPU interconnects and communication libraries such as NCCL
  • Working knowledge of high-performance networking, including InfiniBand and Ethernet
  • Experience with parallel or distributed storage platforms (e.g., VAST, DDN, Lustre, GPFS, NFS) and an understanding of the tradeoffs between serving data from parallel storage and from memory
  • Familiarity with file formats and compression techniques for large datasets, such as LZ4, zstd, time series compression, and columnar storage
  • Experience with workload schedulers such as Slurm, and familiarity with meta-schedulers such as HyperQueue or Ray
  • Proficiency in Python, Bash, and C or C++, with the ability to write automation, tooling, and performance-sensitive code
  • Understanding of how CPU, GPU, networking, and storage interact, and the ability to troubleshoot performance bottlenecks across the stack
  • Proven ability to diagnose and resolve complex performance and reliability issues in distributed systems
  • Background supporting scientific, ML/AI, or other data-intensive workloads
  • Familiarity with feature engineering workflows and their infrastructure demands is a plus

About the company

Susquehanna is a global quantitative trading firm powered by scientific rigor, curiosity, and innovation. Our culture is intellectually driven and highly collaborative, bringing together researchers, engineers, and traders to design and deploy impactful strategies in our systematic trading environment. To meet the unique challenges of global markets, Susquehanna applies machine learning and advanced quantitative research to vast datasets in order to uncover actionable insights and build effective strategies. By uniting deep market expertise with cutting-edge technology, we excel in solving complex problems and pushing boundaries together.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…