Advanced Data Engineer

Your World Recruitment Group

Cardiff, UK

3 days ago

Apply on www.yourworld.com

Prepare application

Role details

Contract type

Temporary to permanent

Employment type

Full-time (> 32 hours)

Compensation

£54,080.0 - £58,240.0

Working hours

Regular working hours

Job source

www.yourworld.com

Tech stack

Airflow Automation of Tests Microsoft Azure Big Data BigQuery Cloud Computing Cloud Engineering Code Review Continuous Integration Information Engineering Data Governance Data Integration

+28 more

Extract Transform Load (ETL) Data Profiling Data Warehousing Relational Databases DevOps Github Python (Programming Language) Meta-Data Management Cloud Services DataOps Cloudera Software Engineering SQL Databases Google Cloud Apache Spark Gitlab Git Containerization Information Technology Data Lineage Production Code Data Management Physical Data Models Hardware Infrastructure Api Design Software Version Control Data Pipelines Docker

Job description

We are seeking an Advanced Data Engineer to design, build, and optimize our cloud-native data architecture and scalable processing pipelines. In this role, you will lead complex data integration projects, drive on-premise to cloud migrations-with a strong focus on Google Cloud Platform (GCP)-and implement modern DataOps practices. You will collaborate closely with multidisciplinary teams to translate business requirements into secure, high-performing data products while upholding software engineering and data governance standards., * Data Engineering & Cloud Architecture: Design, build, test, and maintain scalable ETL/ELT pipelines and production-ready data products. Lead the migration of legacy data platforms, pipelines, and services from on-premises infrastructure to GCP.

Pipeline Orchestration & Optimization: Monitor, troubleshoot, and optimize pipeline performance for large-scale datasets using distributed processing tools (Spark, BigQuery) and workflow orchestrators (Airflow, Cloud Composer, Dagster).
Software & DataOps Best Practices: Apply CI/CD, containerization (Docker), Git-based version control, and automated testing to build production-quality code. Conduct peer code reviews, refactor existing code, and maintain robust technical documentation.
Data Modeling & Governance: Develop conceptual, logical, and physical data models. Implement metadata management, data lineage, data profiling, and validation frameworks to maintain high data quality. Ensure full compliance with UK GDPR, the Data Protection Act 2018, and ethical data guidelines.
Collaboration & Delivery: Work in an Agile environment to translate non-technical business requirements into robust technical solutions. Communicate architectural and data concepts clearly to both technical peers and key business stakeholders.

Requirements

Technical Core: Strong proficiency in SQL and Python; working knowledge or experience with R is highly desirable.
Cloud & Infrastructure: Hands-on experience designing cloud-native solutions (preferably GCP), migrating on-premise systems to the cloud, and managing relational databases/data warehouses.
Data Engineering Stack: Demonstrated experience with big data processing (Spark, Dataproc, BigQuery), orchestration (Airflow/Cloud Composer/Prefect), and API-driven data integration.
DevOps & Quality: Proven background with Git (GitHub/GitLab/Azure DevOps), Docker, CI/CD automation, and modern DataOps principles.
Education & Certifications:
Undergraduate degree in Computer Science, a related technical field, or equivalent practical work experience.
GCP Data Engineer or GCP Cloud Architect certification is highly desirable.