Data Engineer - GCP

Solution IT, Inc.

United States

about 2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Compensation

$135,200.0 - $145,600.0

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Airflow Big Data BigQuery Cloud Computing Cloud Computing Security Cluster Analysis Computer Programming Continuous Integration Data Architecture Data Governance Data Infrastructure Data Security

+21 more

Data Systems Data Warehousing Data Flow Control Identity and Access Management Python (Programming Language) Performance Tuning Scrum Methodology Query Optimization SQL Databases Data Streaming Enterprise Data Management Data Processing Freeform SQL Google Cloud Data Storage Technologies Git Build Management Data Lakes Pyspark Software Version Control Data Pipelines

Job description

We are seeking a highly skilled Data Engineer with strong Google Cloud Platform (GCP) expertise to design and build scalable modern data platforms. The ideal candidate will have hands-on experience implementing Medallion Architecture, building enterprise-grade data pipelines, and optimizing analytical workloads using BigQuery and PySpark., * Design, develop, and maintain scalable batch and real-time data pipelines on GCP

Implement Medallion Architecture (Bronze, Silver, Gold layers) for data processing
Build high-performance data transformations using Python and PySpark
Develop and optimize complex SQL queries for analytical workloads
Work extensively with BigQuery for large-scale data processing and performance tuning
Develop and deploy pipelines using Cloud Dataflow
Orchestrate workflows using Cloud Composer (Apache Airflow)
Manage data storage and lifecycle using Google Cloud Storage (GCS)
Implement version control and CI/CD pipelines using Git-based tools
Ensure data security, governance, and access control using GCP IAM
Optimize data solutions for performance, scalability, reliability, and cost-efficiency

Requirements

Do you have experience in Version control systems?, * Strong hands-on experience with Google Cloud Platform (GCP)

Expertise in BigQuery (partitioning, clustering, query optimization)
Proven experience implementing Medallion Data Architecture
Strong programming skills in Python and PySpark
Advanced proficiency in SQL (complex joins, window functions, performance tuning)
Hands-on experience with Cloud Dataflow
Experience with Cloud Composer (Airflow) for orchestration
Experience working with Google Cloud Storage (GCS)
Knowledge of version control systems (Git) and CI/CD practices
Strong understanding of GCP IAM and cloud security best practices

Preferred Qualifications