Data Platform Engineer

apsa Personnel Concepts GmbH
6 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Shift work
Languages
English
Experience level
Senior

Job location

Tech stack

Big Data
Cloudera Impala
Computer Programming
Linux
Memory Management
Hadoop
Hadoop Distributed File System
Python
Query Optimization
Cloudera
Systems Architecture
Working Model 2D
Apache Zookeeper
Apache Yarn
Spark
Kubernetes
Infrastructure Automation Frameworks
Apache Flink
Kafka
Apache Nifi
Terraform
Serverless Computing

Job description

  • Administration, monitoring, and optimization of a big data platform in the AWS Cloud
  • Management and maintenance of Cloudera On-Cloud PaaS services as well as AWS-native technologies such as Kafka, Flink, NiFi, Iceberg Tables, and DynamoDB
  • Deployment and automation of infrastructure in AWS using Terraform and ArgoCD
  • Execution of updates, upgrades, and incident management for all platform components
  • Advising data platform developers on selecting suitable services for business use cases
  • Taking over 3rd-level support, including monitoring, error analysis, and troubleshooting to ensure a highly available system (including 24/7 on-call duty)
  • Optimization of workloads, e.g.:
  • Flink job memory management
  • Impala and Trino query optimization
  • Evaluation and implementation of cloud-native services in accordance with the existing system architecture

Requirements

  • Very good knowledge of Linux
  • Strong experience with Hadoop services such as HDFS, Zookeeper, Yarn, Impala, Spark, and Kafka
  • Experience with Terraform for infrastructure automation
  • Programming skills in Spark and Python
  • Practical experience with on-call duty and structured troubleshooting
  • Team-oriented working style and strong team spirit
  • Nice to have:
  • Experience with Kubernetes
  • Specific knowledge of Cloudera services such as HDFS, Zookeeper, Yarn, Impala, Hue, Spark, Kafka, Flink, Knox, Ranger RAZ, Streams Messaging Manager, NiFi, and Kudu
  • Experience with rolling restarts in production environments

About the company

Our client, an established company in the software industry, operates a modern, cloud-based data platform in an AWS environment. To strengthen the team, we are looking for an experienced Senior Data Platform Engineer who is passionate about managing, optimizing, and further developing complex big data architectures.

Apply for this position