> Markdown version of [/jobs/ext/2308055-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/2308055-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engineer

- **Company:** Forsyth Barnes
- **Location:** Greater London, UK
- **Experience:** Expert
- **Contract:** Temporary contract
- **Skills:** Adaptable Database Systems, Application Programming Interfaces (APIs), Airflow, Amazon Web Services, Amazon S3, Application Frameworks, Automation of Tests, Big Data, Cloud Database, Continuous Integration, Data Architecture, Data Governance, Data Infrastructure, Data Security, Data Systems, Distributed Computing Environment, File Transfer Protocol (FTP) SSL Extension, Identity and Access Management, Python (Programming Language), Machine Learning, Performance Tuning, SQL Databases, Data Streaming, User-Centered Design, Management of Software Versions, Workflow Management Systems, File Transfer Protocol (FTP), Apache Spark, Cloudformation, Pyspark, Infrastructure Automation Frameworks, Apache Nifi, Data Management, Cloudwatch, Terraform, Data Pipelines, Databricks
- **Published:** August 30, 2026
- **Apply:** https://www.collegerecruiter.com/job/2815459057-lead-data-engineer

## About the Role

* Strong experience as a Senior or Lead Data Engineer with ownership of end-to-end data solutions
 * Expertise in Databricks, PySpark / Spark, SQL, and Python
 * Proven experience building and optimising large-scale data pipelines in production environments
 * Strong knowledge of cloud data architectures, particularly within AWS
 * Experience designing scalable data models and reusable frameworks
 * Hands-on experience with orchestration tools such as Airflow or similar
 * Solid understanding of data governance, lineage, and compliance requirements
 * Experience with CI/CD pipelines and infrastructure as code (e.g. Terraform, CloudFormation)
 * Strong communication skills with the ability to collaborate across technical and non-technical teams, * A hands-on technical leader who can design, build, and deliver solutions independently
 * Someone comfortable working with high-volume, high-throughput data systems
 * Strong problem-solving skills and a pragmatic, delivery-focused mindset
 * Experience mentoring engineers and setting engineering standards and best practices
 * Ability to balance technical excellence with delivery timelines

## Description

In this role, you will help design and evolve a modern Databricks + lakehouse architecture, enabling analytics, machine learning, and investigative teams to generate actionable insights from large-scale datasets., * Own the end-to-end design, build, optimisation, and support of scalable Spark / PySpark data pipelines (batch and streaming)

 * Define and implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality controls
 * Design and manage secure data ingestion frameworks (e.g. Apache NiFi, APIs, SFTP/FTPS) for internal and external data sources
 * Architect and maintain secure AWS-based data infrastructure (S3, IAM, KMS, Glue, Lake Formation, Lambda, Step Functions, CloudWatch, etc.)
 * Implement orchestration using tools such as Airflow, Databricks Workflows, and Step Functions
 * Champion data quality, observability, and reliability (SLAs, monitoring, alerting, reconciliation)
 * Drive CI/CD best practices for data platforms (infrastructure as code, automated testing, versioning, environment promotion)
 * Mentor engineers on distributed data processing, performance optimisation, and cost efficiency
 * Collaborate with data science, product, and compliance teams to translate requirements into scalable data solutions


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)