Data Engineer

iTech US, Inc.

United States

5 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Airflow Amazon Web Services BigQuery Cloud Computing Cloud Database Information Systems Data as a Services Information Engineering Data Governance Extract Transform Load (ETL) Data Masking Relational Databases

+19 more

Software Design Patterns Data Flow Control Python (Programming Language) PostgreSQL Performance Tuning Query Optimization Cloudera SAP NetWeaver Data Management Software Engineering SQL Databases Data Streaming Google Cloud Data Ingestion Change Data Capture Infrastructure Automation Frameworks Information Technology Database Replication Terraform Data Pipelines

Job description

We’re looking for a Data Engineer to design and maintain data pipelines that move and transform data across AWS and Google Cloud Platform environments, with a strong emphasis on PostgreSQL and Cloud SQL. This role centers on building a pipeline that ingests streaming data from AWS Firehose, stages it in GCS, loads it into Cloud SQL (PostgreSQL), applies data masking, and delivers clean data into BigQuery for analytics., * Design, build, and maintain ETL/ELT pipelines using Dataflow, Dataproc, and Airflow (Cloud Composer) for orchestration

Build and manage data ingestion from AWS Firehose into Google Cloud Storage (GCS)
Load and manage relational data in Cloud SQL (PostgreSQL), including schema design, performance tuning, and query optimization
Implement data masking/anonymization using Cloud DLP to protect sensitive data prior to downstream use
Load masked/transformed data into BigQuery for analytics and reporting consumption
Use Datastream for change data capture (CDC) and database replication where applicable
Monitor pipeline health, troubleshoot failures, and optimize for reliability and cost
Collaborate with data analysts, architects, and security/compliance teams to ensure data governance standards are met

Requirements

Hands-on experience with Google Cloud Platform data services: Dataflow, Dataproc, Cloud Composer/Airflow, BigQuery, Cloud SQL, Datastream
Strong PostgreSQL experience schema design, query optimization, administration
Experience with Cloud DLP (or equivalent data masking/anonymization tools)
Experience working with AWS services, particularly Kinesis Data Firehose, and cross-cloud data movement (AWS GCS)
Solid understanding of ETL/ELT design patterns, batch and streaming data processing
Proficiency in Python and/or SQL for pipeline development
Familiarity with data governance, security, and compliance practices (PII handling, masking)

Nice to Have

Experience with Terraform or other IaC tools for provisioning Google Cloud Platform resources
Experience with CI/CD pipelines for data engineering workflows
Background in SAP data extraction/integration

Education : At least a bachelor s degree (or equivalent experience) in Computer Science, Software Engineering, Electronics Engineering, Information Systems, or a closely related field is required for the project