Senior & Principal Data Engineers | Databricks | Outside IR35

Depending on your level, you’ll take ownership of designing, building and/or leading modern Databricks data platforms, working across large-scale data engineering, platform architecture, governance and deployment.

You’ll work with engineering, architecture and business teams to deliver scalable production-grade solutions while helping establish strong engineering standards across the wider team., * Design and build production-grade data pipelines using Databricks, PySpark and Spark SQL

• Design and implement Delta Lake/Medallion Architecture solutions
• Lead technical design across ingestion, transformation and data consumption
• Implement and optimise Unity Catalog for governance, lineage and access control
• Build and maintain CI/CD and Infrastructure-as-Code using Terraform/Databricks Asset Bundles
• Optimise Spark workloads for performance, scalability and cost
• Establish engineering standards around testing, deployment and code quality
• Work closely with architects, engineers and client stakeholders
• Mentor and technically support other engineers
• At Principal level, own platform-level architecture and set technical direction across Databricks workstreams

• Strong commercial Databricks experience
• Advanced PySpark and Spark SQL
• Strong understanding of Spark internals and performance optimisation
• Hands-on Delta Lake and Medallion Architecture experience
• Strong Python engineering skills
• Hands-on Unity Catalog experience
• CI/CD and Infrastructure-as-Code experience
• Terraform and/or Databricks Asset Bundles
• Cloud experience across Azure, AWS or GCP
• Experience delivering large-scale production data platforms
• Strong stakeholder communication skills

For the Principal-level positions, you should also have experience setting technical direction, owning platform architecture and operating as the technical authority across large Databricks engagements.

Desirable

• Databricks Professional-level certifications
• Delta Sharing/Lakehouse Federation
• Structured Streaming/Kafka
• MLflow or AI/ML-adjacent data engineering
• Agentic or AI-driven data engineering patterns
• Previous consultancy or contract delivery experience