GCP Data Architect
Robotics Technologies LLC
Dallas, TX, United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.indeed.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Compensation
$151,840
Working hours
Regular working hours
Job source
Tech stack
Sql Data Warehouse
Active Directory
Application Programming Interfaces (APIs)
Amazon S3
Audit Trail
Build Automation
Big Data
BigQuery
Cloud Storage
Cluster Analysis
Configuration Management
Databases
+37 more
Continuous Integration
Data Architecture
Extract Transform Load (ETL)
Data Transformation
Data Warehousing
Distributed Computing Environment
Data Flow Control
Github
Identity and Access Management
Python (Programming Language)
PostgreSQL
Liquibase
Metadata
Metadata Repositories
MongoDB
Model View Controller (MVC)
Performance Tuning
Query Optimization
Cloudera
Service Layer
SQL Databases
Management of Software Versions
Google Cloud
Data Ingestion
Apache Spark
Git
Event Driven Architecture
Data Lakes
Pyspark
Deployment Automation
Data Analytics
Data Management
Drift Detection
Splunk
Software Version Control
Data Pipelines
Apache Beam
Job description
The IAM Data Modernization project involves migrating an on-premises SQL data warehouse to a target state Data Lake in GCP cloud environment.This modernization establishes a single source of truth for enterprise-wide data-driven decision-making. The IAM Data Modernization program is focused on migrating and modernizing enterprise IAM data platforms from on-premises systems into a scalable, metadata-driven modern data architecture., * Own the end-to-end data architecture for the IAM Data Modernization program across OCP, S3, PostgreSQL metadata repository, and GCP / BigQuery future state architecture.
- Define and guide implementation of a metadata-driven generic ETL framework supporting multiple source types including databases, files, APIs, MongoDB, Splunk, VDS, and Active Directory.
- Architect reusable metadata models using MD_ static metadata tables and OPS_ operational/runtime tables for source configuration, pipeline definition, run tracking, audit logging, checkpointing, schema drift, and notifications.
- Provide architecture direction for MVC-based orchestration, including repository layer, service layer, wrapper, controller, execution plan models, and standardized request/response contracts.
- Design scalable ingestion patterns for full load, incremental load, watermark-based processing, CDC-style patterns, batch ingestion, API ingestion, and file-based ingestion.
- Guide design of S3 data lake zones including Landing, Raw, Archive, and Error zones, with clear promotion, archival, replay, and error-handling patterns.
- Define data quality, schema validation, schema drift detection, CLOB validation, checkpoint restartability, auditability, and operational resilience standards.
- Provide technical leadership for GCP / BigQuery architecture, including S3 to BigQuery integration, schema mapping, dataset design, partitioning, clustering, performance optimization, and secure connectivity.
- Review and guide CI/CD automation using GitHub Actions, JFrog, Harness, Liquibase, Helm, OCP deployments, and environment-specific configuration management.
- Provide technical guidance to team on Python, PySpark, SQL, BigQuery, PostgreSQL, data modeling, performance tuning, and production-ready framework design.
Requirements
- Hands-on experience with PySpark for ETL/ELT, data transformation, and performance optimization
- Solid understanding of distributed data processing concepts
- Strong experience designing data platforms on Google Cloud Platform (GCP)
- Experience with Data Lakes, data warehousing, and large-scale migration programs
- Proven experience designing and implementing data lake architectures (e.g., Bronze/Silver/Gold or layered models).
- Strong knowledge of Cloud Storage (GCS) design, including bucket layout, naming conventions, lifecycle policies, and access controls
- Expertise in partitioning strategies, backfills, and large-scale data organization
- Experience building batch and streaming ingestion pipelines using GCP-native services
- Knowledge of Pub/Sub-based streaming architectures, event schema design, and versioning
- Experience developing scalable batch and streaming pipelines using Dataflow (Apache Beam) and/or Spark (Dataproc)Strong proficiency in BigQuery SQL, including query optimization, partitioning, clustering, and cost control.Experience implementing CI/CD pipelines for data and analytics workloads
- Familiarity with Git-based source control, build automation, and deployment strategies
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.indeed.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦