Databricks Lead Engineer - Hybrid - Contract

Anson McCade

London, UK

6 days ago

Apply on www.totaljobs.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

2 years minimum

Compensation

£149,500.0

Working hours

Regular working hours

Job source

www.totaljobs.com

Tech stack

Artificial Intelligence Continuous Integration Information Engineering Data Masking Data Security Database Development Machine Learning Role-Based Access Control Search Technologies Data Streaming Generative AI Data Lakes

+3 more

Pyspark Machine Learning Operations Databricks

Job description

A hands-on Databricks Lead Engineer is required to lead the delivery of a modern enterprise data platform for a leading financial services organisation.

The role will take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers. Responsibilities will include implementing Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns while mentoring engineers and driving production-quality delivery.

Requirements

At least two years of recent, hands-on Databricks experience
Evidence of delivering Databricks solutions into production environments
An active Databricks Data Engineering, Machine Learning or Generative AI certification
Strong experience implementing Medallion Architecture across Bronze, Silver and Gold layers
Hands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharing
Experience developing Declarative Pipelines with Expectations, including failure handling and observability
Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns
Production-level PySpark and SQL development skills
Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM
Experience with Databricks Workflows, orchestration and CI/CD
Understanding of monitoring, alerting, replay, backfill and operational support strategies
Ability to mentor engineers and act as the technical escalation point
Experience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficial
Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous

Benefits & conditions

Up to £575 per day
Inside IR35
Initial 6-month contract with potential extension
Hybrid working with three days per week on-site in London
ASAP start
Technical leadership of a strategic Databricks implementation
Opportunity to shape engineering standards, reusable patterns and delivery practices