Senior Data Engineer
Hack The Box
Greater London, UK
15 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.collegerecruiter.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
Artificial Intelligence
Airflow
BigTable
BigQuery
Continuous Integration
Data Flow Control
Python (Programming Language)
Machine Learning
Online Analytical Processing
SQL Databases
Data Streaming
Workflow Management Systems
+12 more
Feature Engineering
Snowflake
Build Management
Debezium
Apache Flink
Production Code
Apache Kafka
Spark Streaming
Machine Learning Operations
Vertica
Data Pipelines
Docker
Job description
You will own and evolve our data pipelines on GCP-building new ones, hardening existing ones, improving data quality, and making clean, trustworthy data available across the organisation. You will work end-to-end on streaming and batch pipelines, from CDC and event ingestion through transformation, serving, and the feature layer that powers our ML and AI products., * Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse.
- Help drive the migration off Snowflake onto our GCP native stack and retire legacy pipelines.
- Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates.
- Model data for analytics and for ML-including feature pipelines that serve both training and low-latency online inference.
- Partner with ML engineers on feature stores, drift monitoring and retraining workflows.
- Capture requirements from stakeholders and translate them into well-scoped data products.
- Continuously improve data quality, reliability, observability and cost efficiency.
- Identify new data sources worth acquiring and integrate them cleanly.
Requirements
- Strong data modelling and warehouse architecture skills (dimensional modelling, event-driven, lakehouse patterns).
- Hands-on experience with GCP data services-BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses.
- Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark Structured Streaming, ingesting from Kafka and/or Pub/Sub.
- Solid SQL and strong Python-production-quality code.
- Experience with ClickHouse or another columnar OLAP engine in production.
- Workflow orchestration experience with Airflow (or Prefect/Dagster).
- Comfortable with dbt or equivalent transformation frameworks.
- Experience migrating off legacy warehouses (Snowflake, Redshift, Synapse) onto cloud-native stacks.
- Working knowledge of ML in production-feature engineering, feature stores, model deployment, drift monitoring, retraining.
- Docker and Kubernetes experience.
- CI/CD mindset, infrastructure-as-code sensibility and a bias for simple, observable systems.
- Bonus: CDC tooling (Datastream, Debezium), Vertex AI / Feature Store.
Benefits & conditions
- Private health care.
- Paid paternity leave.
- 25 annual leave days.
- Free lunch & snacks at the office.
- Ticket Restaurant by Edenred.
- Dedicated budget for training and professional development, participation in conferences.
- Full access to our lab offerings for learning how to hack.
- State-of-the-art equipment (Mac, iPhone, mobile plan).
- Flexible WFH (Hybrid Model) - fully remote also available if not in Athens.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.collegerecruiter.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Talks and stories from around this role — technically off-topic, practically not.
Moments
Explore playlistsVideos
See allRelated articles
See all
BB
Benedikt Bischof
about 4 years ago
DS
Dhannush Subramani
Top Big Data Technologies That You Need to Know
about 4 years ago
EM
Eli McGarvie
Data Engineer Salary UK
over 3 years ago
IK
Igor Khokhriakov
How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again
about 1 month ago
CH
Chris Heilmann
Dev Digest 120 - Apple and peers
over 2 years ago
EM
Eli McGarvie
Highest Paying Tech Companies for Developers
over 3 years ago