Data Engineer

gb Robert Walters

Swansea, UK

3 days ago

Apply on www.totaljobs.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.totaljobs.com

Tech stack

Application Programming Interfaces (APIs) Airflow Microsoft Azure Big Data BigQuery Profiling Code Review Computer Programming Continuous Integration Data as a Services Information Engineering Data Governance

+29 more

Data Integration Extract Transform Load (ETL) Data Warehousing Relational Databases DevOps Distributed Computing Environment Github Python (Programming Language) Meta-Data Management Performance Tuning DataOps Cloudera Software Construction Software Engineering SQL Databases Systems Integration Workflow Management Systems Google Cloud Apache Spark Gitlab Git Containerization Information Technology Deployment Automation Data Analytics Data Management Software Version Control Data Pipelines Docker

Job description

Design, build, test and maintain scalable data pipelines and data products.
Develop and optimise ETL/ELT processes for integrating data from multiple sources.
Design and implement cloud-native data solutions, preferably within Google Cloud Platform (GCP).
Support migration of data platforms, pipelines and services from on-premises environments to the cloud.
Develop and maintain data models, data warehouses and data integration solutions.
Implement CI/CD processes and automated deployment pipelines.
Use workflow orchestration tools such as Airflow, Prefect, Dagster or Cloud Composer.
Monitor, troubleshoot and optimise platform and pipeline performance.
Collaborate with technical and non-technical stakeholders to translate business requirements into robust technical solutions.
Contribute to code reviews, testing, technical documentation and software engineering best practices.
Ensure compliance with data governance, security and regulatory requirements, including GDPR.

Requirements

Strong programming skills in Python and SQL (R experience desirable).
Experience with relational databases and data warehousing technologies.
Proven experience building and supporting production-grade ETL/ELT pipelines.
Strong understanding of data modelling principles, including conceptual, logical and physical models.
Experience with Git-based version control systems such as GitHub, GitLab or Azure DevOps.
Experience working in Agile delivery environments.
Experience with cloud platforms, preferably Google Cloud Platform (GCP).
Experience with workflow orchestration tools including Airflow, Prefect, Dagster or Cloud Composer.
Knowledge of containerisation technologies such as Docker.
Experience working with large-scale datasets and distributed processing technologies such as Spark, Dataproc and BigQuery.
Understanding of DevOps and DataOps principles.
Experience integrating systems through APIs and data services.
Strong troubleshooting and performance optimisation skills.

Desirable Skills

Experience with metadata management, data catalogues and lineage tools.
Experience implementing data quality, profiling and validation frameworks.
GCP Data Engineer and/or GCP Architect certification.
Experience within highly regulated or data-driven environments., * Degree in Computer Science, Data Engineering, Software Engineering or a related technical discipline, or equivalent commercial experience.