Cloud Compute Engineer

• Build a shared compute platform: Develop and operate distributed compute infrastructure supporting batch processing, custom jobs, simulation, calibration, preprocessing, and ML training.
• Automate data processing: Build event-driven workflows that automatically process newly ingested data, including flight-test and chamber calibration data.
• Scale compute-intensive workloads: Enable scalable ML training, Foxglove RDL preprocessing, data preparation, replay, and other user-submitted jobs.
• Integrate compute with CI/CD: Enable simulation and regression tests to run automatically in CI/CD pipelines, expanding test coverage beyond standard runner limitations.
• Optimize performance and cost: Profile and improve Python workloads, data flows, memory usage, concurrency, and resource utilization-not just the underlying infrastructure.
• Improve the developer experience: Provide simple interfaces, APIs, or SDKs for engineers to submit, monitor, debug, and rerun jobs without managing infrastructure manually.
• Ensure reliability and observability: Implement logging, metrics, alerting, job traceability, and cost monitoring to make failures easy to diagnose and recover from.
• Enable reproducible workflows: Track code versions, inputs, configurations, and outputs so compute jobs can be reliably reproduced and audited.
• Support large-scale data movement: Design efficient approaches to storage, data transfer, caching, partitioning, and preprocessing.
• Drive adoption across teams: Work directly with engineers to understand their workflows and establish consistent compute patterns across unclassified and classified networks.

• Experience building and operating distributed compute platforms used by multiple engineering teams.
• Strong Python skills, including profiling, concurrency, memory management, and performance optimization.
• Production experience with Ray and Kubernetes.
• Experience designing batch-processing, event-driven, and user-submitted job systems.
• Experience with Azure compute, storage, networking, identity, and monitoring services.
• Demonstrated ability to optimize application code and data pipelines for performance, reliability, and cost.
• Experience implementing observability, automated job recovery, and reproducible processing workflows.
• Ability to build simple developer interfaces and work directly with users to automate complex workflows.

Preferred Skills and Experience

• Strong PyTorch experience, including distributed training, data loading, checkpointing, and GPU utilization.
• Experience scaling ML training across large datasets and distributed compute resources.
• Experience integrating simulation and regression workloads into CI/CD pipelines.
• Familiarity with large-scale engineering datasets, flight-test data, calibration, or scientific computing workflows.
• Experience building compute platforms that support both classified and unclassified environments.
• Experience improving processing throughput and turnaround time through workload optimization and infrastructure scaling.