Senior Data Engineer

CMC Markets UK Plc

London, UK

16 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.totaljobs.com

Tech stack

Airflow Algorithmic Trading Apache HTTP Server Continuous Integration Extract Transform Load (ETL) Linux Distributed Computing Environment Python (Programming Language) PostgreSQL Operational Databases Reference Data Standard Sql

+17 more

SQL Databases Parquet Data Storage Technologies Snowflake Apache Spark Git KDB+ Data Lakes Dask Apache Kafka Data Management Machine Learning Operations Video Streaming Vertica Data Pipelines Docker Databricks

Job description

Build scalable ETL/ELT pipelines for batch and streaming market data.
Develop production-grade Python and SQL solutions.
Ingest, normalise and validate market data, including trades, ticks, Level 1 & 2 quotes, order books, reference data and corporate actions.
Implement automated data quality, reconciliation and monitoring frameworks.
Optimise data storage using Parquet, Arrow and modern data lake technologies.
Partner with quantitative researchers, traders and engineering teams to deliver trusted datasets.

Requirements

Strong Python and SQL.
Experience building scalable, production data pipelines.
Expertise with Parquet and modern data storage formats.
Deep understanding of high-frequency market data, including tick data, order books and market data normalisation.
Experience handling time-series challenges such as timestamp precision, sequence gaps, duplicate and out-of-order events.
Experience with distributed processing (Spark, Polars, Dask, Ray or similar).
Experience with Kafka or similar streaming technologies.
Familiarity with PostgreSQL, ClickHouse, Snowflake, Databricks, kdb+ or similar.
Docker, Linux, Git and CI/CD experience.

Highly Desirable

Familiarity with OneTick, PostgreSQL, ClickHouse, Snowflake, Databricks, kdb+ or similar.
Order book reconstruction
Tick-to-bar aggregation
Airflow, Dagster, Prefect or MLflow
Experience supporting quantitative research or trading systems
Multi-asset market data (Equities, Futures, FX, Options, ETFs, CFDs)

What We’re Looking For

The successful candidate will have a strong background in market data engineering with proven experience building reliable, scalable data platforms.

Our non-negotiable requirements are:

Expert Python
Strong SQL
Scalable data pipeline development
Apache Parquet
Market data normalisation
Automated data quality controls
Deep tick and market data expertise, Experience in MLOps is advantageous but not essential. We welcome candidates with strong experience in either market data engineering or MLOps, provided they have the skills to build and operate production-grade data platforms.