GCP Data Engineer

The Senior Cloud Data Engineer will serve as the technical anchor for the physical data migration and modern pipeline engineering required to build DLA’s GCP Data Fabric. This role will drive the critical transition of moving over 100 terabytes of uncompressed business data from distributed Cloudera Hadoop (HDFS/Hive) clusters and AWS S3/FSx data dumps into a serverless, optimized storage layer in BigQuery.

The Data Engineer will establish secure Medallion Architecture pathways (Bronze * Silver *Gold) using modern GCP data movement frameworks.

Key Roles and Responsibilities

• BigQuery Data Lakehouse Engineering: Design, implement, and maintain scalable BigQuery data warehouses. Map legacy HDFS/Hive schemas and parquet/flat-file configurations directly into BigQuery tables.
• ETL Reduction & SQL Modernization: Lead the massive migration effort to reverse-engineer and collapse legacy data pipelines. Analyze the 87% data-wrangling/ETL script footprint in DLA’s codebase and translate it into high-performance, native BigQuery SQL, Dataflow (Apache Beam), Dataproc (Serverless Spark), or dbt pipelines.
• Authoritative Source Integration via SLT & Datastream: Architect secure ingestion pipelines to synchronize transactional tables directly from S/4HANA (EBS) database layers to GCP using SAP SLT via Pub/Sub and Private Google Access (PGA). Deploy Google Datastream for continuous Change Data Capture (CDC) replication from legacy non-SAP engines (Oracle 19c, IBM DB2 on z/OS).
• Storage Performance Benchmarking & Optimization: Optimize BigQuery scan overhead and combat potential project bill shock by implementing Column Clustering and Partitioning strategies customized around target Mission Support Element (MSC) high-frequency join/filter columns. Conduct performance benchmarking on GCP Filestore deployments to satisfy the required 125 MB/s per core throughput metric for legacy file transformations.
• Data Governance & Zero-Trust Persistence: Enforce structural file lifecycle management rules and fine-grained column-level and row-level access controls, utilizing Dataplex and BigQuery data masking parameters to satisfy stringent DoD IL5/IL6 security standards.

• Google Core Data Stack: Expert-level mastery of BigQuery (Dremel Engine, SQL, IAM controls), Dataflow, Dataproc, Cloud Composer (Airflow), Data Fusion, and Pub/Sub.
• Distributed Computing Expertise: Advanced experience working with Apache Spark, Hive Metastore, PySpark pipelines, and data migration utilities (DistCp, Storage Transfer Service).
• Languages: Superior proficiency in Python, PySpark, and advanced ANSI SQL.
• Certifications: Google Cloud Professional Data Engineer (Highly Preferred); Must maintain DoD 8570 baseline compliance (e.g., Security+ CE)

Pay: $45.00 - $50.00 per hour