Senior Data Engineer - Agents Systems

Kraken
Greater London, UK
18 days ago
Apply on www.collegerecruiter.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
2 years minimum
Working hours
Regular working hours

Tech stack

Artificial Intelligence Information Engineering Data Governance Interaction Design Python (Programming Language) Machine Learning SQL Databases AI Infrastructure Feature Engineering Backend Build Management Apache Flink
+4 more
Apache Kafka Machine Learning Operations Stream Processing Data Pipelines

Job description

The Agent Systems team is a 0*1 engineering group building internal AI-powered agents that interact directly with company systems. The mandate is to dramatically increase internal speed of execution through automation, intelligent inference, and workflow orchestration.

This team operates at the intersection of AI, backend systems, and applied product engineering. The work is pragmatic and fast-moving - prototypes are expected, but the bar for production reliability remains high. Engineers on this team move fluidly between experimentation and shipping, building systems that reason over user interactions and take meaningful action across internal tools.

You’ll work closely with cross-functional partners across product, infrastructure, and internal operations to deploy agent-driven capabilities that compound leverage across the organization.

The opportunity

  • Own and evolve streaming data pipelines that power live inference and real-time model serving across Kraken’s AI infrastructure
  • Design and build feature stores that serve low-latency, high-reliability features to production ML models
  • Implement and maintain streaming systems using RisingWave, Apache Flink, or Kafka Streams, selecting the right tool for the workload
  • Partner with ML engineers and AI infra teams to define data contracts, feature schemas, and pipeline SLAs
  • Drive pipelines toward real-time where batch exists today reducing latency from hours to seconds
  • Ensure data quality, observability, and auditability across all streaming and feature engineering systems
  • Contribute to inference pipeline tooling where data engineering and model serving intersect
  • Evaluate emerging streaming and feature store technologies and shape the team’s technical roadmap

Requirements

  • 5+ years in data engineering with at least 2 years focused on streaming systems in production
  • Hands-on experience with RisingWave, Apache Flink, Kafka Streams, or comparable stream processing frameworks
  • Strong understanding of feature store design - online/offline consistency, point-in-time correctness, low-latency serving
  • Experience building data pipelines that feed production ML models or inference systems
  • Proficiency in Python and/or Scala; SQL fluency required
  • Familiarity with data quality frameworks, pipeline observability, and SLA ownership
  • Comfortable operating in a fast-moving, ambiguous environment embedded within an AI-focused team, * Direct experience with RisingWave in production
  • Exposure to inference pipeline architecture or model serving infrastructure
  • Experience with feature platforms
  • Crypto or fintech domain experience

About the company

Payward - the parent company behind Kraken, NinjaTrader, Breakout, xStocks, Payward Services and CF Benchmarks - has spent the last 15 years building one of the most modern and globally accessible financial infrastructure platforms in the industry, built to advance an open, global financial system., Founded in 2011, Kraken is one of the world’s longest-standing crypto platforms, trusted by over 10 million individuals and institutions across the globe. It offers spot trading, margin, futures, staking, and OTC services, with products built for both individual investors and institutional clients.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.collegerecruiter.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:55 min

Infrastructure challenges when combining Kafka with Apache Flink

Bobur Umurzokov · LIVE

1:37 min

Introduction to Apache Kafka benchmarking and performance analysis

Kirill Kulikov · LIVE

1:52 min

Structuring and scaling the backend engineering team

Stefan Lingler Stefan Lingler +1 · Coffee With Developers

50 sec

Identifying real-world applications for stream processing technology usage

Soroosh Khodami Soroosh Khodami · World Congress 2024

1:09 min

Evaluating mature stream processing frameworks for production systems

Soroosh Khodami Soroosh Khodami · World Congress 2024

2:04 min

Comparing offline data analytics with online stream processing

Artem Volk Artem Volk +1 · World Congress 2024

Videos

See all

Related articles

See all