Kubernetes Engineer

• Apache Spark optimization to support thousands of concurrent jobs processing large volumes of Bigdata. (candidates are working in smaller environments)
• Active contributions to open-source Kubernetes, Karpenter, or Spark projects
• Configure persistent volume claims and provision container-native storage using Amazon EBS CSI driver for block storage and Amazon EFS CSI driver for shared file access, We are seeking an expert Kubernetes engineer to configure, deploy, and operate mission-critical Amazon EKS infrastructure supporting petabyte-scale data processing workloads. This role requires deep technical expertise in Kubernetes internals, large-scale cluster management, and Apache Spark optimization to support thousands of concurrent jobs processing large volumes of Bigdata.

Core Responsibilities

• Design, deploy, and maintain production-grade Amazon EKS clusters architected for petabyte-scale data processing with high availability and fault tolerance
• Operate in air-gapped private VPC environments without internet access, managing secure package repositories and container registries
• Debug and resolve complex distributed systems issues across EKS, Karpenter, and Spark, including scheduling bottlenecks, node scaling delays, and cascading failures under heavy load
• Implement Karpenter consolidation and disruption policies balancing cost optimization with job resiliency
• Manage spot and on-demand instance strategies with robust node interruption handling
• Define and enforce ResourceQuotas, LimitRanges, and PriorityClasses to ensure fair resource distribution and prevent resource starvation
• Configure persistent volume claims and provision container-native storage using Amazon EBS CSI driver for block storage and Amazon EFS CSI driver for shared file access
• Optimize storage configurations for resiliency, high-throughput data processing workloads
• Implement logging, alerting, and anomaly detection to identify provisioning failures, executor loss, and throughput degradation before broader system impact
• Design fault-tolerant architectures with retry strategies, checkpointing, and graceful degradation patterns minimizing re-computation on failure
• Develop and manage configs in a private VPC without internet access

• Kubernetes certification (CKA, CKAD, or CKS) & AWS certifications
• Active contributions to open-source Kubernetes, Karpenter, or Spark projects
• Familiarity with FinOps practices and cost optimization at scale
• Background in data engineering or analytics platforms