> Markdown version of [/jobs/ext/50671-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/50671-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** Betfred - **Location:** Manchester, UK - **Experience:** Expert - **Salary:** £55,000.0 - £80,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Amazon Web Services, Amazon S3, Apache HTTP Server, Batch Processing, Data as a Services, Information Engineering, Data Governance, Data Infrastructure, Extract Transform Load (ETL), Data Masking, Distributed Systems, Fault Tolerance, Identity and Access Management, Python (Programming Language), Data Streaming, Transaction Data, User-Centered Design, Real Time Systems, Delivery Pipeline, Amazon Virtual Private Cloud (VPC), Event Driven Architecture, Pyspark, Apache Flink, Real Time Data, Apache Kafka, Data Delivery, Terraform, Stream Processing - **Published:** May 16, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=a95054950682621e ## About the Role Do you have experience in Terraform?, * Streaming & Messaging: Hands-on experience with Apache Kafka (Amazon MSK) and Apache Flink. * Platform & AWS Proficiency: Strong background in AWS Platform Engineering. You should be comfortable with IAM roles, VPC networking for data services, and managing infrastructure via Terraform. * Advanced Technical Stack: Proven mastery in Python/Java (for Flink/Kafka custom UDFs). * Expertise in PySpark and Apache Iceberg for transactional data lake management. * Distributed Systems Design: Demonstrated ability to design fault-tolerant systems. You understand the trade-offs between latency, throughput, and correctness in a distributed environment. * Architectural Vision: Experience moving organisations from legacy ETL patterns to modern Event-Driven Architectures (EDA). * Data Governance & Security: Practical experience implementing encryption-at-rest/transit within Kafka, schema registry management, and GDPR-compliant data masking in real-time streams. ## Description We are seeking a highly experienced Senior Data Engineer to be a lead in the architecture and evolution of our real-time data ecosystem. In this role, you will be a primary driver for our next-generation streaming platform, moving beyond traditional batch processing to embrace low-latency, event-driven architectures. Built predominantly on AWS and utilising Flink, Kafka (MSK), and Iceberg, PySpark, our infrastructure is designed for massive scalability and "fresh" data delivery. You will support bridging the gap between Data Engineering and Platform Engineering, ensuring our streaming clusters are not only high-performing but also automated, observable, and resilient. You will mentor the team in streaming best practices and set the gold standard for real-time systems., * Architect Real-Time Streaming Solutions: Lead the end-to-end design of stateful and stateless stream processing applications using Apache Flink and Apache Kafka. Optimise consumers, producers, and stream-to-stream joins for high throughput and exactly-once processing. * Infrastructure as Code & Platform Engineering: Take a "Platform-first" approach by automating the provisioning and scaling of data infrastructure. Utilise Terraform, to manage AWS resources (MSK, EMR, Glue) and implement robust CI/CD pipelines for data applications.Modern Lakehouse Evolution: Drive the technical strategy for our Iceberg-based lakehouse, focusing on real-time ingestion patterns that bridge the gap between Kafka and S3/Redshift. * Observability & Reliability: Define and implement enterprise-level monitoring for streaming health (lag, backpressure, state-size) and enforce data quality frameworks that validate data in flight.Cross-Functional Technical Leadership: Collaborate with Data Scientists to operationalise feature stores and real-time ML inference pipelines, ensuring data is available in milliseconds, not hours. * Performance Engineering: Proactively identify and resolve complex bottlenecks in distributed systems, such as Kafka partition imbalances, Flink checkpointing issues, or EMR resource contention. * Mentorship: Lead "Deep Dive" sessions on streaming theory (watermarks, windowing, state management) and provide hands-on guidance to engineers transitioning from batch to stream. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Infrastructure as Code: The Developer's Secret Weapon](https://www.wearedevelopers.com/videos/1221-infrastructure-as-code-the-developer-s-secret-weapon) - [WeAreDevelopers LIVE - CSS is DOOMed](https://www.wearedevelopers.com/videos/1838-wearedevelopers-live-css-is-doomed) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)