Data Engineer (REMOTE) (remote)
Role details
Job location
Tech stack
Job description
Infrastructure Deployment & Management: Efficiently deploy and manage infrastructure on Google Cloud Platform (GCP) including network architectures (Shared VPC, Hub-and-Spoke), security implementations (IAM, Secret Manager, firewalls, Identity-Aware Proxy), DNS configuration, VPN, and Load Balancing. # Data Processing & Transformation: Utilize Hadoop cluster with Hive for querying data, and PySpark for data transformations. Implement job orchestration using Airflow. # Core GCP Services Management: Work extensively with services like Google Kubernetes Engine (GKE), Cloud Run, BigQuery, Compute Engine, and Composer, all managed through Terraform. # Application Implementation: Develop and implement Python applications for various GCP services. # Integrate and manage GitLab Magenta CI/CD pipelines for automating cloud deployment, testing, and configuration of diverse data pipelines. # Implement comprehensive security measures, manage IAM policies, secrets using Secret Manager, and
Requirements
enforce identity-aware policies. # Data Integration: Handle integration of data sources from CDI, Datendrehscheibe (FTP servers), TARDIS API´s and Google Cloud Storage (GCS). # Multi-environment Deployment: Create and deploy workloads across Development (DEV), Testing (TEST), and Production (PROD) environments. # Implement AI solutions using Google's Vertex AI for building and deploying machine learning models. # Must be a certified GCP Cloud Architect or Data Engineer. Strong proficiency in Google Cloud Platform (GCP) Expertise in Terraform for infrastructure management Skilled in Python for application implementation Experience with GitLab CI/CD for automation Deep knowledge of network architectures, security implementations, and management of core GCP services Proficiency in employing data processing tools like Hive, PySpark , and data orchestration tools like Airflow Familiarity with managing and integrating diverse data sources Certified GCP Cloud Architect and Data Engineer Hybrid work model (telework/face-to-face). Continuous training. Please send CV in English.