Data Platform Engineer

Xcede
Greater London, UK
6 days ago
Apply on www.collegerecruiter.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours

Tech stack

Artificial Intelligence Amazon Web Services Amazon Elastic Compute Cloud Amazon S3 Data Analysis BigQuery Information Engineering Data Infrastructure Extract Transform Load (ETL) Python (Programming Language) Standard Sql SQL Databases
+7 more
Snowflake Kubernetes Apache Kafka Functional Programming Terraform Stream Processing Databricks

Job description

A strong opportunity for an engineer who wants to build the data platform an entire organisation runs on, not just the pipelines on top of it.

We’re partnering with a well-backed B2B SaaS business whose software underpins millions of transactions a year. Several established products now sit under one platform, and the company is putting serious investment behind the data infrastructure that ties it all together. It’s a business with genuine scale and stability but the pace and ambition of something far younger, and it’s going through a period of real growth and change.

With a data engineering team already in place, the focus now is on building the platform layer beneath them: the foundation, tooling and reusable building blocks that let everyone else move data quickly and safely. The platform is already on a modern footing, so this is about extending what it can do rather than untangling what came before.

The Role

You’ll help scale the Databricks lakehouse that powers analytics and AI across the business, stand up the infrastructure behind new data commercialisation products, and design the abstractions and templates that let other teams work with data consistently from ingestion through to access. It’s a hands-on role with real ownership: you’ll take work from design through to production and have a genuine say in how the platform and its engineering practices evolve.

What you’ll be doing

  • Scaling and supporting the data lakehouse, building new capability that serves analytics and AI use cases across the business
  • Designing the blueprints, abstractions and reusable templates that let service and analytics teams handle data safely and consistently
  • Building the infrastructure and logic behind new data commercialisation products, working with product and commercial teams to turn data into revenue
  • Writing ETL and contributing to data modelling where the platform alone isn’t enough, for both internal analytics and external-facing products
  • Owning your work end to end, from first design through to deployment
  • Helping shape engineering standards and better ways of working for the team’s internal customers

What we’re looking for

  • Strong Python and a track record of building data infrastructure that other engineers and analysts depend on
  • Hands-on AWS and Terraform, with infrastructure as code as your default way of working
  • Experience building on a modern lakehouse or warehouse (Databricks ideally, though Snowflake or BigQuery travels well)
  • Ingestion and transformation tooling such as Fivetran and dbt
  • A platform mindset: you take as much pride in building the tools and paved roads other engineers rely on as you do in shipping your own pipelines

Nice to have

  • Working knowledge of streaming systems such as Kafka, and a feel for the abstractions that make event-driven data easy for other teams to consume
  • Solid SQL and a good sense for data modelling, even if neither is your daily focus
  • Awareness of how data workloads run in production on ECS and Kubernetes

The Stack

Core

  • Python and SQL
  • Databricks, with lakehouse storage on S3
  • AWS (EventBridge, Kinesis, Lambda, S3, EC2) with Terraform for infrastructure as code

Supporting

  • dbt and Fivetran for transformation and ingestion
  • Kafka for streaming
  • ECS and Kubernetes for orchestration

Don’t worry if you don’t tick every box. If you’ve got a solid data engineering foundation and the appetite to build, we’d like to hear from you.

If this sounds like you and you’d like to hear more (or explore other roles), apply here or reach me directly at . Feel free to attach a CV for review.

Requirements

  • Strong Python and a track record of building data infrastructure that other engineers and analysts depend on
  • Hands-on AWS and Terraform, with infrastructure as code as your default way of working
  • Experience building on a modern lakehouse or warehouse (Databricks ideally, though Snowflake or BigQuery travels well)
  • Ingestion and transformation tooling such as Fivetran and dbt
  • A platform mindset: you take as much pride in building the tools and paved roads other engineers rely on as you do in shipping your own pipelines, * Working knowledge of streaming systems such as Kafka, and a feel for the abstractions that make event-driven data easy for other teams to consume
  • Solid SQL and a good sense for data modelling, even if neither is your daily focus
  • Awareness of how data workloads run in production on ECS and Kubernetes, * Python and SQL
  • Databricks, with lakehouse storage on S3
  • AWS (EventBridge, Kinesis, Lambda, S3, EC2) with Terraform for infrastructure as code, * dbt and Fivetran for transformation and ingestion
  • Kafka for streaming
  • ECS and Kubernetes for orchestration

Don’t worry if you don’t tick every box. If you’ve got a solid data engineering foundation and the appetite to build, we’d like to hear from you.

About the company

A strong opportunity for an engineer who wants to build the data platform an entire organisation runs on, not just the pipelines on top of it.

We’re partnering with a well-backed B2B SaaS business whose software underpins millions of transactions a year. Several established products now sit under one platform, and the company is putting serious investment behind the data infrastructure that ties it all together. It’s a business with genuine scale and stability but the pace and ambition of something far younger, and it’s going through a period of real growth and change.

With a data engineering team already in place, the focus now is on building the platform layer beneath them: the foundation, tooling and reusable building blocks that let everyone else move data quickly and safely. The platform is already on a modern footing, so this is about extending what it can do rather than untangling what came before.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.collegerecruiter.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

1:34 min

Essential commands for running and testing Terraform configurations

Hennie Francis · LIVE

3:43 min

The enduring legacy of the amazon S3 storage API

Chris Heilmann +3 · LIVE

2:30 min

Leveraging BigQuery ML for scalable SQL-based segmentation experiments

Julian Joseph · LIVE

2:32 min

Overview of Terraform and Terraform Cloud features

Devlin Duldulao · LIVE

3:44 min

Automating storage savings with S3 intelligent tiering

Sébastien Stormacq · World Congress 2021

Videos

See all

Related articles

See all