Data Engineer - Databricks - Outside IR35 - £450 - £525

Primus Connect

London, UK

2 days ago

Apply on www.careerboard.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Compensation

£117,000.0 - £136,500.0

Working hours

Regular working hours

Job source

www.careerboard.com

Tech stack

Agile Methodology Airflow Amazon Web Services Microsoft Azure Code Review Data Validation Information Engineering Data Infrastructure Extract Transform Load (ETL) Apache Hive Python (Programming Language) Data Streaming

+9 more

Apache Spark Git Data Lakes Pyspark Apache Kafka Spark Streaming Terraform Data Pipelines Databricks

Job description

This is a hands-on engineering position where you will take ownership of data pipeline development, work with modern Databricks technologies and collaborate closely with senior engineers and wider delivery teams., * Design, build and maintain ETL/ELT data pipelines using Databricks, PySpark and Spark SQL

Develop and optimise Delta Lake tables using Bronze, Silver and Gold medallion architecture
Ingest data from a variety of source systems into Databricks
Write clean, maintainable and well-tested Python/data engineering code
Implement data quality checks, validation and monitoring
Participate in code reviews and collaborative development
Contribute to CI/CD pipelines for data engineering workloads
Work with senior engineers to deliver scalable data platform solutions
Collaborate with technical, analytics and business stakeholders
Participate in Agile ceremonies and delivery activities

Requirements

They are expanding their engineering capability and are looking for a Mid-Level Data Engineer with strong commercial Databricks and Spark experience to join an active delivery team., * Commercial experience building data pipelines using Databricks

Strong PySpark and Spark SQL skills
Solid Python experience within data engineering
Practical experience with Delta Lake
Understanding of medallion architecture and modern data platform principles
Experience working with Azure, AWS or GCP
Good understanding of Git and collaborative development practices
Experience working within Agile delivery environments
Strong communication skills and the ability to work effectively in a remote team

Desirable Experience

Databricks certification
Unity Catalog
Databricks Asset Bundles
Terraform/Infrastructure as Code
Databricks Workflows
Airflow
Spark Structured Streaming
Kafka or Azure Event Hubs

About the company

Primus Connect is working with a growing Data & AI Consultancy that is delivering modern data platforms and Databricks solutions for a range of enterprise clients.