DATA Engineer - GCP

Krest Global Solutions LLC

Atlanta, GA, United States

4 days ago

Apply on www.juju.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Compensation

$120,640.0 - $124,800.0

Working hours

Regular working hours

Job source

www.juju.com

Tech stack

Application Programming Interfaces (APIs) Artificial Intelligence Airflow BigQuery Cloud Storage Cluster Analysis Continuous Integration Data Architecture Information Engineering Data Governance Data Infrastructure Data Integrity

+24 more

Extract Transform Load (ETL) Data Transformation Data Warehousing Data Flow Control Data Intelligence Python (Programming Language) Performance Tuning Cloudera SQL Databases Google Cloud Snowflake Apache Spark Containerization Kubernetes Infrastructure Automation Frameworks Information Technology Apache Kafka Machine Learning Operations Terraform Data Pipelines Automation Anywhere Docker Jenkins Databricks

Job description

Data Pipeline Development: Build scalable batch and real-time pipelines using Dataflow, Pub/Sub, Cloud Composer, and Dataproc.
Data Warehousing: Design and optimize analytical models in BigQuery, implementing best practices in schema design and performance tuning.
Infrastructure & CI/CD: Deploy data infrastructure with Terraform; create and manage CI/CD pipelines for workflow automation.
Data Quality & Governance: Implement validation checks, ensure data integrity, and enforce security and governance practices.
AI/ML Integration: Collaborate with data scientists to support Vertex AI workflows and explore the use of Gemini models (via BigQuery ML or Vertex AI APIs) for advanced data transformation.
Cross-Team Collaboration: Work with analysts, scientists, and business stakeholders to deliver impactful data solution.

Requirements

We are looking for a highly capable Data Engineer to design, build, and maintain reliable, scalable data pipelines and infrastructure on Google Cloud Platform (GCP).. The ideal candidate is proficient with BigQuery, Dataflow, and has experience integrating modern tools like Vertex AI and Gemini models for intelligent data workflows., * Strong experience with GCP services: BigQuery, Dataflow, Pub/Sub, Cloud Storage

Expertise in SQL, Python, and ETL/ELT development
Knowledge of Infrastructure as Code tools (e.g., Terraform)
Familiarity with CI/CD tools: Google Cloud Build, Jenkins
Understanding of data modeling, partitioning, clustering, and materialized views
Working knowledge of data quality frameworks and governance principles
Experience with Vertex AI, or a strong interest in ML/AI workflows on GCP
Apache Spark, Kafka, Apache Airflow
DBT or Dataform for transformations
Docker, Kubernetes for Containerization
Snowflake, Databricks or other modern platforms *

Soft Skills:

Strong communication and collaboration skills
Ability to manage priorities and work independently
Analytical thinking and problem-solving mindset, * Google Cloud certifications (e.g., Professional Data Engineer, ML Engineer)
Domain experience in energy, utilities, or industrial sectors, * Education: Bachelor’s degree in computer science, Engineering, Statistics, or related technical field (or equivalent experience)
Professional Experience:
8-10 years of experience in data engineering
Proven hands-on experience with GCP and modern data architecture