GCP Data Engineer

Wolverhampton

Wolverhampton, UK

2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

apply4u.co.uk

Tech stack

Agile Methodology Airflow Business Analytics Applications BigQuery Cloud Storage Continuous Integration Data Transformation Data Vault Modeling DevOps Digital Assets Data Flow Control Identity and Access Management

+17 more

Python (Programming Language) Mainframes Standard Sql Service Development Studio Software Engineering Teradata SQL Scripting Google Cloud Data Ingestion Informatica Powercenter Integration Tests Kubernetes Data Lineage Star Schema Software Version Control Data Pipelines Legacy Systems

Job description

Key ResponsibilitiesPipeline Development: Design, develop, and maintain robust end-to-end data pipelines and orchestration workflows using Python, Apache Airflow, and Cloud Composer on GCP.Data Modernization: Bridge the gap between legacy/on-premise database environments and modern cloud architectures, ensuring seamless data ingestion and migration.Optimization & Performance: Architect and tune analytics solutions at scale within BigQuery, utilizing partitioning, clustering, and materialized views to maximize efficiency and control costs.CI/CD & Software Best Practices: Embed rigorous software engineering principles, including Git/GitHub version control, strict PR disciplines, automated unit/integration testing, and deployment via CI/CD pipelines.Data Governance & Compliance: Embed data quality controls, access management, data lineage, and privacy frameworks required to meet stringent prudential/regulatory banking compliance standards.Collaboration: Partner with cross-functional Agile teams, including Product Owners, DevOps Engineers, and Business SMEs, to translate complex technical requirements into business-ready assets.

Requirements

Required Technical Skills & ExperienceGCP Core Stack: Proven production-grade experience building and managing cloud data solutions on Google Cloud Platform (specifically BigQuery, Cloud Composer/Airflow, Cloud Storage, and Dataflow).Programming & Scripting: Strong hands-on coding capability in Python for data engineering, service development, and pipeline automation.Data Transformation & Modeling: Proficient in SQL, data modeling (Star Schema, Data Vault, normalisation), and data transformation frameworks like dbt.Regulated Environment Experience: Prior experience working within UK financial services, banking, or a similarly heavily regulated framework handling sensitive/critical data assets is highly preferred.Containerization (Desirable): Experience or familiarity with containerizing and operating services using Dockerand Kubernetes (GKE).Legacy Systems (Desirable): Familiarity with enterprise legacy platforms (e.g., Informatica, Teradata, Mainframe) to assist with migration patterns