Data Engineer - GCP

The ideal candidate is a hands-on engineer with strong experience building batch and real-time streaming pipelines, developing reusable Python data frameworks, optimizing large-scale workloads, and designing high-performance data models in BigQuery. The engineer will help establish reliable, secure, observable, and cost-efficient data solutions using modern GCP architecture and engineering practices., n \n

• Design, develop, and maintain enterprise-scale batch and streaming data pipelines using Python, PySpark, Dataflow, Dataproc, Pub/Sub, BigQuery, and Cloud Composer \n

• Build modular, reusable, and testable Python frameworks for data ingestion, transformation, validation, and pipeline automation \n

• Develop high-performance data-processing solutions using PySpark, Spark SQL, and Apache Beam \n

• Integrate data from relational and NoSQL databases, REST APIs, flat files, cloud storage, third-party platforms, and event streams \n

• Build real-time streaming pipelines using Pub/Sub and Dataflow, including windowing, triggers, watermarking, late-arriving data, deduplication, and delivery semantics \n

• Design event-driven architectures supporting real-time analytics, operational reporting, and downstream data products \n

• Create and maintain workflow orchestration using Cloud Composer and Apache Airflow, including reusable operators, dependency management, retries, alerting, and recovery \n

• Design scalable BigQuery schemas using dimensional, normalized, denormalized, and lakehouse modeling patterns \n

• Optimize BigQuery workloads through effective partitioning, clustering, materialized views, query tuning, and cost-management strategies \n

• Tune Spark workloads by resolving data skew, excessive shuffling, inefficient joins, memory constraints, partitioning issues, and small-file problems \n

• Develop incremental loading, change data capture, and high-volume data-processing solutions \n

• Implement automated data-quality checks, reconciliation, schema validation, lineage, logging, monitoring, and alerting \n

• Develop Python-based data services and APIs using frameworks such as FastAPI \n

• Apply security best practices using IAM, service accounts, encryption, secret management, and least-privilege access \n

• Build and maintain CI/CD pipelines for data applications using Cloud Build, GitHub Actions, or equivalent tools \n

• Provision and manage GCP data infrastructure using Terraform \n

• Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable data solutions \n

• Create technical documentation, data mappings, operational runbooks, and architecture diagrams \n

• Participate in code reviews and promote Python, SQL, GCP, and data-engineering best practices \n

We are seeking a highly skilled GCP Data Engineer with 5+ years of experience designing and building scalable, production-grade data platforms. This role requires deep expertise in Google Cloud Platform, Python, PySpark, SQL, and distributed data processing across the complete data lifecycle-including ingestion, transformation, orchestration, storage, quality, and serving., * 5+ years of hands-on experience in data engineering, including building production-grade data pipelines and platforms \n

• Strong programming expertise in Python, including object-oriented programming, reusable modules, exception handling, logging, testing, packaging, and performance optimization \n

• Advanced experience with PySpark, Spark SQL, and Apache Spark \n

• Strong knowledge of GCP data services, particularly BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Cloud Storage \n

• Advanced SQL skills, including complex joins, window functions, CTEs, query optimization, and large-scale data transformation \n

• Deep expertise in BigQuery, including schema design, partitioning, clustering, performance tuning, workload management, and cost optimization \n

• Hands-on experience building batch and streaming pipelines with Dataflow and Apache Beam using Python \n

• Strong experience designing and operating Spark workloads on Dataproc \n

• Experience developing and managing Airflow DAGs using Cloud Composer \n

• Experience designing Pub/Sub topics, subscriptions, schemas, retry policies, dead-letter queues, and deduplication strategies \n

• Strong understanding of distributed processing concepts, including partitioning, shuffling, parallelism, fault tolerance, and delivery guarantees \n

• Solid understanding of Spark internals, including execution plans, DAGs, Catalyst optimization, memory management, and Spark UI troubleshooting \n

• Experience with dimensional data modeling, including star and snowflake schemas and SCD Type 1 and Type 2 \n

• Experience processing large-scale partitioned datasets and implementing incremental data loads \n

• Experience with data formats such as Parquet, Avro, ORC, JSON, and Delta \n

• Strong understanding of streaming concepts, including event time, processing time, windowing, triggers, watermarking, and exactly-once versus at-least-once processing \n

• Experience implementing data-quality, observability, reconciliation, and production-monitoring frameworks \n

• Proficiency with Git, automated testing, Agile delivery, and CI/CD practices \n, n \n

• Experience building Python APIs and microservices using FastAPI or Flask \n

• Working knowledge of Terraform and infrastructure-as-code practices \n

• Experience with CDC technologies and patterns \n

• Familiarity with Dataform or dbt for SQL-based data transformations \n

• Knowledge of GCP security, networking, IAM, Secret Manager, and data-governance practices \n

• Google Cloud Professional Data Engineer certification is preferred \n