Lead Data Engineer

Our client, a leading technology firm, is currently seeking an experienced Lead Data Engineer to join their team in London. This is a permanent, full-time position.

About the Opportunity

The successful candidate will architect, build, and scale the data infrastructure powering an advanced AI-driven platform. This hands-on leadership role involves designing scalable systems from the ground up, playing a critical role in building the data foundations that support AI products, analytics capabilities, machine learning workflows, and large-scale operational systems. You will work closely with AI engineers, backend developers, and product teams to ensure data is processed efficiently, reliably, and securely, while helping to define the future direction of the organisation’s data architecture. As a senior member of the team, you will establish best practices, influence technical strategy, and contribute to the growth of the data function., * Architect and build scalable data pipelines and infrastructure to support AI and product systems.

• Design and maintain data ingestion, transformation, and storage architectures for operational and AI workloads.
• Develop and manage batch and real-time data processing pipelines.
• Build and optimise systems for vector search, data retrieval, and machine learning workflows.
• Ensure data reliability, security, governance, and compliance across the platform.
• Collaborate closely with AI and backend engineering teams to support model training, inference, and product development.
• Implement monitoring, observability, and data quality frameworks.
• Optimise the performance of large-scale datasets, data warehouses, and query systems.
• Contribute to technical architecture decisions and long-term data strategy.
• Mentor engineers and help establish engineering standards, processes, and best practices.

• Experience in Data Engineering, Software Engineering, or Backend Engineering.
• Proven experience designing and building scalable data pipelines and distributed data systems.
• Strong experience with relational databases, preferably PostgreSQL.
• Experience working with NoSQL databases.
• Experience with vector databases utilised in modern AI applications.
• Strong Python development skills.

Data & Infrastructure:

Experience with some of the following technologies is highly desirable:

Frameworks & Infrastructure:

• Apache Spark
• Apache Airflow
• Kafka
• Elasticsearch / OpenSearch

Databases:

• PostgreSQL
• MongoDB
• Vector databases such as Qdrant, Milvus, or pgvector

Python Ecosystem:

• Pandas
• Polars

Engineering Skills:

• Experience building highly scalable backend systems.
• Strong understanding of data modelling and storage architecture.
• Deep knowledge of data processing, optimisation, and performance tuning.
• Experience implementing reliable and maintainable data engineering practices.

Nice to Have:

• Experience with JavaScript and Node.js.
• Experience working on AI, machine learning, or generative AI platforms.
• Familiarity with stream-processing and event-driven architectures.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Previous experience within a high-growth startup or scale-up environment.

• Opportunity to shape the data architecture of a growing AI-driven platform.
• Significant ownership and influence over technical direction.
• Exposure to modern AI, machine learning, and data engineering technologies.
• Collaborative environment where innovation and impact are highly valued.
• Strong career growth potential within an ambitious and scaling business.