High-Performance Computing Consultant III

NSF | NCAR - Computational and Information Systems Lab (CISL) is seeking a part-time, term High-Performance Computing Consultant III. This individual will provide technical consulting and software engineering expertise on the use, performance optimization, and application development for the Computational and Information Systems Laboratory (CISL)'s multi-vendor High-Performance Computing (HPC) & High-Performance Storage (HPS) environments and large-scale scientific application codes. The consultant is expected to have in-depth experience in designing and developing software for relevant application domains in scientific computing, demonstrated expertise in the tools and technologies applied in HPC including the development and maintenance of the HPC user environment, and to apply this experience to Consulting Service Group (CSG) activities. This consultant will assist in CISL's system procurements and planning and contribute technical leadership, including leading team efforts to resolve difficult application and system problems, contributing as a subject-matter expert for user documentation. The HPC Consultant is a strong advocate for the needs and best interests of the user community. The HPC Consultant strongly contributes to the High-Performance Computing Division (HPCD) and CISL leadership teams for architecture, vendor relationships, and general policy direction as pertains to the use and usability of CISL's high-performance computing and high-performance storage resources, including leadership of special projects related to the user-facing environment., This position will likely work ~10 hours week for a 6-month term. There is a possibility of term renewal based on funding availability and performance., * Application Performance and Optimization: Profiles and evaluates performance of user applications or benchmark codes to ensure efficient use of CISL resources, and may provide code modifications as required. Debugs and profiles parallel programs (primarily Fortran, often in C/C++, Python, Matlab, and R), especially MPI and OpenMP/OpenACC applications, including general purpose graphical processing unit (GPGPU) scientific applications.

• Technical Issue Resolution: Technical Issue Resolution: Serves as a reach-back technical expert within CSG to resolve difficult technical problems with the CISL multi-vendor HPC and storage production environment, which often requires working with vendors, NCAR and CISL software engineers, HPC systems engineers, and scientists in a team environment, exchanging ideas and providing programming support.
• User Support: Provides expert-level consulting in a "tiered" help desk environment as a reach-back resource available to other full-time consulting staff.
• Software Installation and Development: Builds and maintains local software packages and provides expert compiler support for the user-facing HPC software environment through collaboration with full-time consulting staff
• System Testing and Benchmarking: Participates in creating, enhancing or running benchmark test suite of applications after system software installations and upgrades and for acceptance testing of new systems.
• Documentation: Gathers technical information and supports the development of user documentation on complex topics such as parallel compilers, profilers, and debuggers.

• Master's degree in computational science, atmospheric science, or engineering, and extensive and progressive relevant experience, which is typically gained by eight to twelve years of work experience.

Knowledge, Skills, and Abilities:

• Proven experience with programs composed in C/C++ and/or Fortran 90 programming languages.
• Proven experience with the OpenACC programming paradigms and/or the CUDA parallel computing platform.
• Proven experience with multiple Linux-based or Unix-based operating systems.
• Demonstrated knowledge of GPGPU application development.
• Proven experience with MPI and knowledge of advanced MPI concepts such as MPI-IO and hybrid MPI+Threads.
• Demonstrated experience with OpenMP and OpenACC directive-based programming paradigms.
• Experience supporting multi-user Python installations for HPC and scientific applications.
• Familiarity with building and maintaining software packages using versioning systems, makefiles and shell scripting
• Experience developing software in a team environment with software engineering tools and processes.
• Familiarity with one or more tools for plotting scientific data, including Matlab, R, NCAR Graphics, NCL, or IDL.
• Demonstrated skill in applying software analysis, design, and development methodologies to computing applications and/or programming problems.
• Experience developing software for more than one scientific application domain.
• Ability to evaluate, interpret, and apply new technical knowledge.
• Skill in building and maintaining software packages using versioning systems, makefiles and shell-scripting.
• Skill with upgrading the hardware and software stack on HPC and HPS systems
• Demonstrated skill and good judgment in working with a variety of people including users, colleagues, vendors, and experts external to UCAR, in a problem-solving context.
• Extreme reliability in timely follow-up with users, vendors, and support teams.
• Excellent oral and written communication skills in English.
• Demonstrated skill in developing and making short presentations on complicated computing topics.

DESIRED (but not required):

• Experience running or working with one or more of NCAR's flagship codes-for example, CESM or WRF-or other large application codes.
• Familiarity with the technological challenges associated with Artificial Intelligence / Machine Learning (AI/ML) workflows, particularly in a HPC environment.
• Experience developing software in a team environment with software engineering tools and processes.
• Experience with parallel development tools such as profilers, optimizers, and debuggers.
• Knowledge of numerical analysis or differential equations.
• Skill with HPC and HPS networking, high-performance interconnects
• Skill with HPC and HPS benchmarking
• Knowledge of and experience researching innovative high-performance computing, storage, file systems, Cloud, and tape archive technologies
• Skill with analyzing user and data workflows and developing enhanced user and data transfer workflows

• Tuition Assistance, time off allowance to attend classes, and other professional development opportunities.
• UCAR contributes 10% of your eligible pay into your retirement account; 100% fully vested on day one.
• Starting minimum accrual of 20 days of personal time off each year (prorated for less than full-time positions).
• 10 paid holidays.
• 12 weeks of paid parental leave.
• Short-term medical leave paid at 100% of your regular salary.
• EcoPass for local Colorado residents to use the Denver and Boulder-area transit system at no cost.

UCAR affirms its commitment to employees through competitive benefits. In addition to medical, dental, vision, retirement, and life insurance, UCAR offers a variety of programs focused on work-life balance and professional, and personal development. These include, ChatGPT and similar AI software are powerful tools that are changing the way society receives, processes, and leverages information promptly. While we acknowledge its benefits and do not restrict leveraging it with job applications, we highly encourage a majority of the applicant material to be original work.