Senior Data Engineer - Databricks

Anson McCade

London, UK

1 day ago

Apply on www.totaljobs.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Compensation

£149,500.0

Working hours

Regular working hours

Job source

www.totaljobs.com

Tech stack

Unity 3d Amazon Web Services Microsoft Azure Cloud Computing Cloud Engineering Software Quality Continuous Integration Data Architecture Extract Transform Load (ETL) Data Security Python (Programming Language) Performance Tuning

+12 more

Cloud Services SQL Databases Technical Data Management Systems Apache Spark Git Data Lakes Pyspark Data Management Software Version Control Data Pipelines Databricks Programming Languages

Job description

Lead the design and implementation of scalable data architectures on the Databricks Lakehouse platform.
Oversee the development of efficient ETL/ELT pipelines using PySpark, SQL, and Delta Lake.
Mentor and guide junior data engineers, promoting best practices in code quality, CI/CD, and performance tuning.
Collaborate with data architects, analysts, and stakeholders to translate business requirements into technical data solutions.
Ensure data security, governance, and quality standards are maintained across all data workflows.
Attend the London office for 3 days per week for collaborative working and stakeholder engagement.

Requirements

We are seeking an experienced Databricks Lead Engineer to drive the design, development, and optimization of enterprise-scale data platforms. The ideal candidate will have strong technical leadership capabilities, deep expertise in the Databricks Lakehouse platform, and a proven track record of delivering robust data pipelines and architecture in complex environments., * Databricks Expertise: Extensive hands-on experience with Databricks, Delta Lake, Unity Catalog, and Spark optimization.

Programming Languages: Advanced proficiency in Python/PySpark and SQL.
Cloud & Infrastructure: Strong background in cloud data platforms (Azure, AWS, or GCP) and cloud-native architectures.
Engineering Best Practices: Experience with CI/CD pipelines, version control (Git), and infrastructure-as-code.
Leadership: Demonstrated experience leading engineering teams, managing workstreams, and driving technical delivery.