Lead Data Engineer

Understanding Recruitment
Greater London, UK
21 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Compensation
£100,000.0
Working hours
Regular working hours

Tech stack

Artificial Intelligence Amazon Web Services Microsoft Azure Data Architecture Extract Transform Load (ETL) Python (Programming Language) Machine Learning Standard Sql Software Deployment SQL Databases Google Cloud Cloud Platform System
+5 more
Apache Spark Pyspark Virtual Agents Data Pipelines Databricks

Job description

Lead Data Engineer - London (Hybrid)

A fast-growing AI and data consultancy that helps global businesses unlock insights from consumer data using cloud-native platforms, machine learning and agentic AI are looking for a Lead Data Engineer to help shape both the technology and the engineering team.

What you’ll be doing:

  • Lead the design and delivery of enterprise-scale data and AI platforms
  • Build cloud-native solutions using Databricks, Azure, AWS and GCP
  • Develop scalable ETL/ELT pipelines with Python, PySpark and SQL
  • Define engineering standards and mentor a team of Data & Platform Engineers
  • Work on AI deployment, orchestration and monitoring in production
  • Collaborate with clients, architects, data scientists and product teams to deliver high-impact solutions

What we’re looking for:

  • Strong experience building cloud-based data platforms
  • Expertise with Python, SQL, Spark/PySpark and Databricks
  • Experience designing scalable data architectures and modern data pipelines
  • Previous technical leadership or mentoring experience
  • Azure, AWS or GCP experience

What’s in it for you?

  • Salary up to £100,000
  • Generous pension contribution, private healthcare, personal learning budget etc.
  • Clear progression and exposure to cutting-edge AI and agentic technologies

Apply now for immediate consideration!

Requirements

  • Strong experience building cloud-based data platforms
  • Expertise with Python, SQL, Spark/PySpark and Databricks
  • Experience designing scalable data architectures and modern data pipelines
  • Previous technical leadership or mentoring experience
  • Azure, AWS or GCP experience

Benefits & conditions

  • Salary up to £100,000
  • Generous pension contribution, private healthcare, personal learning budget etc.
  • Clear progression and exposure to cutting-edge AI and agentic technologies

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.understandingrecruitment.com

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

4:06 min

Using ClickHouse as a foundation for fast analytics

Hellmar Becker Hellmar Becker · WWC Europe 2026

2:27 min

Managing traffic and tracking costs with Databricks Unity Catalog

Viktoria Semaan Viktoria Semaan · WWC Europe 2026

4:32 min

Harnessing Spark with Python using PySpark and Py4J

Ayon Roy · LIVE

3:14 min

Structuring career paths and localized data architectures

Ulrich Wurstbauer +1 · LIVE

1:56 min

Creating immutable blockchain tables using standard SQL syntax

Wei Hu Wei Hu · WWC 2023

Videos

See all

Related articles

See all