Senior Data Engineer

WILLOW VENTURES
London, UK
13 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Compensation
£95,000.0 - £97,000.0
Working hours
Regular working hours
Job source

Tech stack

Java (Programming Language) Application Programming Interfaces (APIs) Airflow Amazon Web Services Data Analysis Microsoft Azure Backup Devices Big Data BigQuery Cloud Computing Cloud Database Continuous Integration
+52 more
Data Architecture Information Engineering Data Governance Data Infrastructure Extract Transform Load (ETL) Data Security Dataspaces Data Systems Data Warehousing Dimensional Modeling Disaster Recovery Github Apache Hadoop Identity and Access Management Python (Programming Language) PostgreSQL Machine Learning Meta-Data Management Microsoft SQL Server MongoDB MySQL NoSQL Oracle (Applications) Power BI Cloud Services Scala (Programming Language) Software Engineering SQL Databases Tableau (Software) Talend Enterprise Data Management Qliksense Google Cloud Enterprise Software Applications Azure Data Factory Informatica Powercenter Snowflake Apache Spark Database Performance Gitlab Git Data Lakes Infrastructure Automation Frameworks Information Technology Cassandra AWS Glue Star Schema Apache Kafka Looker Analytics Data Pipelines Databricks Programming Languages

Job description

As a Senior Data Engineer, you will work closely with data scientists, analytics teams, software engineers, and business stakeholders to build scalable data pipelines, develop cloud-based data solutions, and ensure the reliability, security, and integrity of enterprise data assets. You will play a critical role in implementing modern data architecture, supporting business intelligence initiatives, and driving continuous improvements across the organization’s data ecosystem., * Design, develop, and maintain scalable data pipelines that support enterprise reporting, analytics, and machine learning initiatives.

  • Build and optimize ETL and ELT processes for ingesting, transforming, and loading data from multiple structured and unstructured sources.
  • Design and manage modern data warehouses, data lakes, and lakehouse architectures.
  • Develop cloud-native data solutions using Microsoft Azure, Amazon Web Services (AWS), or Google Cloud Platform (GCP).
  • Collaborate with business stakeholders to understand data requirements and deliver high-quality data solutions.
  • Optimize database performance, SQL queries, and large-scale data processing workflows.
  • Implement robust data quality, validation, governance, and monitoring processes to ensure data accuracy and consistency.
  • Develop scalable data models that support business intelligence, reporting, and advanced analytics.
  • Build automated workflows using orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Glue.
  • Work closely with software engineering teams to integrate enterprise applications, APIs, and cloud platforms with data infrastructure.
  • Implement security controls, access management, backup strategies, and disaster recovery processes for enterprise data platforms.
  • Troubleshoot complex data engineering issues, perform root cause analysis, and implement long-term solutions.
  • Mentor junior data engineers and promote engineering best practices across the data team.
  • Evaluate emerging technologies and recommend innovative solutions that improve scalability, reliability, performance, and operational efficiency.

Requirements

The successful candidate will possess strong technical expertise in cloud technologies, data engineering frameworks, and database management while demonstrating excellent analytical and problem-solving skills., * Bachelor’s degree in Computer Science, Data Engineering, Information Technology, Mathematics, Engineering, or a related discipline, or equivalent professional experience.

  • Significant experience in data engineering, data platform development, or enterprise data architecture.
  • Strong proficiency in SQL and programming languages such as Python, Scala, Java, or Spark.
  • Hands-on experience building ETL and ELT pipelines using Azure Data Factory, AWS Glue, Apache Airflow, Informatica, Talend, or similar tools.
  • Experience with cloud platforms including Microsoft Azure, Amazon Web Services (AWS), or Google Cloud Platform (GCP).
  • Strong knowledge of relational and NoSQL databases such as SQL Server, PostgreSQL, MySQL, Oracle, MongoDB, Cassandra, Snowflake, or BigQuery.
  • Experience working with big data technologies including Apache Spark, Hadoop, Kafka, Databricks, or Delta Lake.
  • Strong understanding of dimensional modelling, data warehousing concepts, Star Schema, Snowflake Schema, and data lake architectures.
  • Experience using Git, Azure DevOps, GitHub, GitLab, CI/CD pipelines, and Infrastructure as Code practices.
  • Knowledge of data governance, GDPR, data security, metadata management, and enterprise data quality frameworks.
  • Experience supporting business intelligence platforms such as Power BI, Tableau, Looker, or Qlik Sense.
  • Excellent analytical, communication, stakeholder management, and problem-solving skills.
  • Professional certifications such as Microsoft Certified: Azure Data Engineer Associate, AWS Certified Data Engineer - Associate, Google Professional Data Engineer, Databricks Data Engineer Associate, or equivalent are highly desirable.
  • Eligibility to work in the United Kingdom.

Benefits & conditions

Pulled from the full job description

  • Financial planning services
  • Gym membership
  • Life insurance
  • Free parking
  • On-site gym
  • Casual dress
  • Health & wellbeing programme

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on uk.indeed.com

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

6:21 min

Investigating push inefficiencies with upstream Git experts

Jonathan Creamer · Coffee With Developers

2:18 min

Scaling MySQL databases for massive user growth

Johannes Nicolai Johannes Nicolai +1 · LIVE

2:37 min

Comparing traditional SQL tables versus NoSQL non-tabular databases

Stanimira Vlaeva · JS Congress

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

56 sec

Favorite git commands and the importance of patch commits

Eileen Uchitelle Eileen Uchitelle +1 · Coffee With Developers

Videos

See all

Related articles

See all