Senior Data Engineer
Role details
Job location
Tech stack
Job description
We are seeking a highly experienced Senior Data Engineer to be a lead in the architecture and evolution of our real-time data ecosystem. In this role, you will be a primary driver for our next-generation streaming platform, moving beyond traditional batch processing to embrace low-latency, event-driven architectures.
Built predominantly on AWS and utilising Flink, Kafka (MSK), and Iceberg, PySpark, our infrastructure is designed for massive scalability and "fresh" data delivery. You will support bridging the gap between Data Engineering and Platform Engineering, ensuring our streaming clusters are not only high-performing but also automated, observable, and resilient. You will mentor the team in streaming best practices and set the gold standard for real-time systems., * Architect Real-Time Streaming Solutions: Lead the end-to-end design of stateful and stateless stream processing applications using Apache Flink and Apache Kafka. Optimise consumers, producers, and stream-to-stream joins for high throughput and exactly-once processing.
- Infrastructure as Code & Platform Engineering: Take a "Platform-first" approach by automating the provisioning and scaling of data infrastructure. Utilise Terraform, to manage AWS resources (MSK, EMR, Glue) and implement robust CI/CD pipelines for data applications.Modern Lakehouse Evolution: Drive the technical strategy for our Iceberg-based lakehouse, focusing on real-time ingestion patterns that bridge the gap between Kafka and S3/Redshift.
- Observability & Reliability: Define and implement enterprise-level monitoring for streaming health (lag, backpressure, state-size) and enforce data quality frameworks that validate data in flight.Cross-Functional Technical Leadership: Collaborate with Data Scientists to operationalise feature stores and real-time ML inference pipelines, ensuring data is available in milliseconds, not hours.
- Performance Engineering: Proactively identify and resolve complex bottlenecks in distributed systems, such as Kafka partition imbalances, Flink checkpointing issues, or EMR resource contention.
- Mentorship: Lead "Deep Dive" sessions on streaming theory (watermarks, windowing, state management) and provide hands-on guidance to engineers transitioning from batch to stream.
Requirements
Do you have experience in Terraform?, * Streaming & Messaging: Hands-on experience with Apache Kafka (Amazon MSK) and Apache Flink.
- Platform & AWS Proficiency: Strong background in AWS Platform Engineering. You should be comfortable with IAM roles, VPC networking for data services, and managing infrastructure via Terraform.
- Advanced Technical Stack: Proven mastery in Python/Java (for Flink/Kafka custom UDFs).
- Expertise in PySpark and Apache Iceberg for transactional data lake management.
- Distributed Systems Design: Demonstrated ability to design fault-tolerant systems. You understand the trade-offs between latency, throughput, and correctness in a distributed environment.
- Architectural Vision: Experience moving organisations from legacy ETL patterns to modern Event-Driven Architectures (EDA).
- Data Governance & Security: Practical experience implementing encryption-at-rest/transit within Kafka, schema registry management, and GDPR-compliant data masking in real-time streams.
Benefits & conditions
We offer a variety of competitive benefits, some of which vary depending on the role you're recruited to. Some of what you can expect in this role includes:
A competitive rate of pay and pension contribution ( £55,000 - £80,000)
Generous discretionary bonus schemes, incentives and competitions
An annual leave entitlement that increases with length of service
Access to an online GP 24/7, 365 days a year for you and your immediate family.
Employee wellbeing support through our Employee Assistance Programme
Enhanced Maternity & Paternity Pay
Long Service Recognition
Access to a pay day savings scheme, financial coach and up to 40% of your earned wage ahead of payday, through Wagestream.