Senior Data Engineer

We are seeking a highly experienced Senior Data Engineer to be a lead in the architecture and evolution of our real-time data ecosystem. In this role, you will be a primary driver for our next-generation streaming platform, moving beyond traditional batch processing to embrace low-latency, event-driven architectures.

Built predominantly on AWS and utilising Flink, Kafka (MSK), and Iceberg, PySpark, our infrastructure is designed for massive scalability and “fresh” data delivery. You will support bridging the gap between Data Engineering and Platform Engineering, ensuring our streaming clusters are not only high-performing but also automated, observable, and resilient. You will mentor the team in streaming best practices and set the gold standard for real-time systems., * Architect Real-Time Streaming Solutions: Lead the end-to-end design of stateful and stateless stream processing applications using Apache Flink and Apache Kafka. Optimise consumers, producers, and stream-to-stream joins for high throughput and exactly-once processing.

• Infrastructure as Code & Platform Engineering: Take a “Platform-first” approach by automating the provisioning and scaling of data infrastructure. Utilise Terraform, to manage AWS resources (MSK, EMR, Glue) and implement robust CI/CD pipelines for data applications.Modern Lakehouse Evolution: Drive the technical strategy for our Iceberg-based lakehouse, focusing on real-time ingestion patterns that bridge the gap between Kafka and S3/Redshift.
• Observability & Reliability: Define and implement enterprise-level monitoring for streaming health (lag, backpressure, state-size) and enforce data quality frameworks that validate data in flight.Cross-Functional Technical Leadership: Collaborate with Data Scientists to operationalise feature stores and real-time ML inference pipelines, ensuring data is available in milliseconds, not hours.
• Performance Engineering: Proactively identify and resolve complex bottlenecks in distributed systems, such as Kafka partition imbalances, Flink checkpointing issues, or EMR resource contention.
• Mentorship: Lead “Deep Dive” sessions on streaming theory (watermarks, windowing, state management) and provide hands-on guidance to engineers transitioning from batch to stream.

Do you have experience in Terraform?, * Streaming & Messaging: Hands-on experience with Apache Kafka (Amazon MSK) and Apache Flink.

• Platform & AWS Proficiency: Strong background in AWS Platform Engineering. You should be comfortable with IAM roles, VPC networking for data services, and managing infrastructure via Terraform.
• Advanced Technical Stack: Proven mastery in Python/Java (for Flink/Kafka custom UDFs).
• Expertise in PySpark and Apache Iceberg for transactional data lake management.
• Distributed Systems Design: Demonstrated ability to design fault-tolerant systems. You understand the trade-offs between latency, throughput, and correctness in a distributed environment.
• Architectural Vision: Experience moving organisations from legacy ETL patterns to modern Event-Driven Architectures (EDA).
• Data Governance & Security: Practical experience implementing encryption-at-rest/transit within Kafka, schema registry management, and GDPR-compliant data masking in real-time streams.

We offer a variety of competitive benefits, some of which vary depending on the role you’re recruited to. Some of what you can expect in this role includes:

A competitive rate of pay and pension contribution ( £55,000 - £80,000)

Generous discretionary bonus schemes, incentives and competitions

An annual leave entitlement that increases with length of service

Access to an online GP 24/7, 365 days a year for you and your immediate family.

Employee wellbeing support through our Employee Assistance Programme

Enhanced Maternity & Paternity Pay

Long Service Recognition

Access to a pay day savings scheme, financial coach and up to 40% of your earned wage ahead of payday, through Wagestream.

Our mission is to dominate the betting and gaming industry on a global scale and we need the very best Tech talent to help us achieve this.

We recently migrated all of our customers onto our very own proprietary platform - so it’s an exciting time to join us. With the help of our new platform, we’re able to pioneer new products and drive more advanced, creative technologies. The result? Unrivalled experiences for millions of customers worldwide.

Betfred’s Technology department is driven by innovation, and you’ll be at the heart of unlocking our new platform’s potential. So, if you want to help shape the future of betting and gaming, then it’s to time to join us.