> Markdown version of [/jobs/ext/4088923-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/4088923-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** Betfred - **Location:** Manchester, United Kingdom - **Experience:** Expert - **Salary:** £85,000 - £105,000 - **Contract:** Permanent contract - **Skills:** Agile Methodology, Amazon Web Services, Amazon S3, Computing Platforms, Code Review, Cyber Security, Continuous Integration, Customer Data Management, Information Engineering, Data Migration, Data Warehousing, Python (Programming Language), Machine Learning, Microsoft SQL Server, Software Architecture, SQL Databases, Data Streaming, Cloud Platform System, AWS Lambda, Event Driven Architecture, Build Management, Data Lakes, Pyspark, Apache Flink, Modeling and Simulation, AWS Glue, AWS Data Analytics, Real Time Data, Apache Kafka, Spark Streaming, Terraform, Stream Processing, Data Pipelines, Amazon Elastic Mapreduce (EMR), Amazon Redshift - **Published:** October 27, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=01801bffb793368b ## About the Role Deep, hands-on expertise across core AWS data services such as AWS Glue, AWS Lambda, AWS S3, AWS Redshift, and Amazon EMR, with the ability to make and defend architectural decisions. Advanced proficiency in Python and SQL, including PySpark for large-scale batch and streaming processing, and strong command of data modelling and data warehousing concepts. Production experience designing and operating streaming architectures with Apache Kafka or Amazon MSK, including stream processing with Spark Structured Streaming. Hands-on experience with open table formats, such as Apache Iceberg or Delta Lake, including partitioning, schema evolution and table maintenance at scale. Experience with pipeline orchestration and infrastructure-as-code using Terraform. Experience acting as technical lead on a significant data platform or migration programme, setting standards others follow. Track record of mentoring and developing engineers through review, pairing and design guidance. Ability to diagnose and resolve complex, ambiguous data issues, and to teach others to do the same. Ability to effectively communicate with both technical and non-technical stakeholders, including senior leadership. Experience with data security and privacy best practices, and how to build them into platform design. Experience with Agile development methodologies. Desirable: Experience with Apache Flink for stateful stream processing would be desirable. Experience in migrating data from on-premises systems, such as SQL Server, to cloud environments would be desirable. Experience in a regulated industry, and awareness of how data platform design intersects with regulatory, compliance and responsible gambling obligations, would be an advantage. ## Description This Lead Data Engineer will be the senior technical authority for our data platform, setting the architectural direction and engineering standards for the team. This hands-on role will focus on leading the technical delivery of our migration from SQL Server to an AWS lakehouse, and designing the real-time streaming architecture that underpins automation, business intelligence, and analytics, while mentoring engineers across the team., Own the technical architecture, standards and design patterns for the data platform, covering batch and streaming pipelines built on AWS Glue, AWS Lambda, AWS S3 and Redshift. Lead the technical delivery of the Data Warehouse migration from SQL Server to AWS, including target modelling, cutover approach, reconciliation strategy and technical risk management. Design and build the real-time data architecture using Amazon MSK / Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly-once semantics and stream-to-lakehouse integration. Define and embed data quality, validation and observability frameworks, and hold the team to them through design and code review. Govern the lakehouse and data warehouse design, ensuring consistency, scalability and sensible cost characteristics across the platform. Own platform infrastructure-as-code (Terraform) and CI/CD standards, so environments and pipelines are reproducible and reviewable. Monitor and optimise AWS platform spend, making the cost and performance trade-offs explicit when proposing architecture. Act as the technical escalation point for complex data issues and performance bottlenecks, leading root-cause analysis and driving permanent fixes. Collaborate with data scientists, analysts and other engineering leads to support the deployment of machine learning and advanced analytical solutions. Mentor and technically coach data engineers through pairing, review and design sessions, and contribute to technical assessment during hiring. Represent data engineering in cross-functional architecture, planning and prioritisation discussions, translating technical trade-offs for senior technical and non-technical stakeholders. Embed data security and privacy by design, working with Information Security and our Data Protection team on how customer data is modelled, accessed and retained.