Lead Data Engineer

Understanding Recruitment
Greater London, UK
3 days ago
Apply on www.understandingrecruitment.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours

Tech stack

Query Performance Artificial Intelligence Airflow Data Analysis Big Data Cloud Computing Data Architecture Information Engineering Data Infrastructure Distributed Data Store Elasticsearch Python (Programming Language)
+13 more
NoSQL Software Product Management Azure Machine Learning Search Technologies Apache Spark Backend Event Driven Architecture Storage Technologies Real Time Data Apache Kafka Machine Learning Operations Data Pipelines Automation Anywhere

Job description

Lead Data Engineer

Want to own the data architecture behind an AI product while the foundations are still being built?

This is an early-stage lead role where you’ll shape the systems powering real-time data, AI workflows, vector search and analytics - with the freedom to make the key technical decisions.

What’s in it for you?

  • Own the data architecture and technical direction
  • Build the data backbone of an AI-first product from the ground up
  • Work closely with AI, backend and product engineers
  • Solve problems across streaming, vector search and ML data infrastructure
  • Help define tooling, standards and the future data team
  • Competitive salary + equity
  • Join early enough to have genuine influence over how things are built

What you’ll be doing

  • Architecting scalable batch and real-time data pipelines
  • Designing ingestion, transformation and storage systems
  • Building infrastructure for vector search, retrieval and ML workflows
  • Improving data quality, observability and reliability
  • Optimising large-scale datasets and query performance
  • Supporting AI training, inference and product features
  • Making long-term architecture and tooling decisions
  • Helping grow and mentor the data engineering function

What you’ll bring

  • Strong experience in data engineering or backend engineering
  • Experience designing scalable pipelines and distributed data systems
  • Strong Python skills
  • Experience with relational and NoSQL databases
  • Hands-on experience with vector databases
  • Good understanding of data modelling, performance and storage architecture
  • Experience with technologies such as Spark, Airflow, Kafka or Elasticsearch/OpenSearch

Experience with AI/ML platforms, cloud infrastructure or event-driven architectures would be useful, but you don’t need everything on the list.

Requirements

  • Strong experience in data engineering or backend engineering
  • Experience designing scalable pipelines and distributed data systems
  • Strong Python skills
  • Experience with relational and NoSQL databases
  • Hands-on experience with vector databases
  • Good understanding of data modelling, performance and storage architecture
  • Experience with technologies such as Spark, Airflow, Kafka or Elasticsearch/OpenSearch

Experience with AI/ML platforms, cloud infrastructure or event-driven architectures would be useful, but you don’t need everything on the list.

Benefits & conditions

Want to own the data architecture behind an AI product while the foundations are still being built?

This is an early-stage lead role where you’ll shape the systems powering real-time data, AI workflows, vector search and analytics - with the freedom to make the key technical decisions.

What’s in it for you?

  • Own the data architecture and technical direction
  • Build the data backbone of an AI-first product from the ground up
  • Work closely with AI, backend and product engineers
  • Solve problems across streaming, vector search and ML data infrastructure
  • Help define tooling, standards and the future data team
  • Competitive salary + equity
  • Join early enough to have genuine influence over how things are built

What you’ll be doing

  • Architecting scalable batch and real-time data pipelines
  • Designing ingestion, transformation and storage systems
  • Building infrastructure for vector search, retrieval and ML workflows
  • Improving data quality, observability and reliability
  • Optimising large-scale datasets and query performance
  • Supporting AI training, inference and product features
  • Making long-term architecture and tooling decisions
  • Helping grow and mentor the data engineering function

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.understandingrecruitment.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

2:37 min

Comparing traditional SQL tables versus NoSQL non-tabular databases

Stanimira Vlaeva · JS Congress

2:15 min

Empowering domain teams with an open data platform

Sandhya Menon Sandhya Menon · World Congress 2026 Europe

1:52 min

Structuring and scaling the backend engineering team

Stefan Lingler Stefan Lingler +1 · Coffee With Developers

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

3:16 min

Terminology differences between relational and NoSQL databases

Tim Faulkes · LIVE

3:14 min

Structuring career paths and localized data architectures

Ulrich Wurstbauer +1 · LIVE

Videos

See all

Related articles

See all