> Markdown version of [/jobs/ext/2041120-senior-principal-data-engineers-databricks-outside-ir35](https://www.wearedevelopers.com/jobs/ext/2041120-senior-principal-data-engineers-databricks-outside-ir35). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Senior & Principal Data Engineers | Databricks | Outside IR35

- **Company:** Primus Connect
- **Location:** London, UK (Remote available)
- **Experience:** Expert
- **Salary:** £143,000.0 - £182,000.0
- **Contract:** Temporary to permanent
- **Skills:** Artificial Intelligence, Amazon Web Services, Computing Platforms, Microsoft Azure, Big Data, Cloud Computing, Software Quality, Continuous Integration, Information Engineering, Apache Hive, Python (Programming Language), Operational Databases, Performance Tuning, Data Streaming, Apache Spark, IT Architecture, Build Management, Data Lakes, Pyspark, Apache Kafka, Data Management, Machine Learning Operations, Terraform, Data Pipelines, Databricks
- **Published:** August 13, 2026
- **Apply:** https://www.careerboard.com/pt/en/find-jobs-in-United-Kingdom/-D1ABD97FC195E75735/

## About the Role

* Strong commercial Databricks experience
 * Advanced PySpark and Spark SQL
 * Strong understanding of Spark internals and performance optimisation
 * Hands-on Delta Lake and Medallion Architecture experience
 * Strong Python engineering skills
 * Hands-on Unity Catalog experience
 * CI/CD and Infrastructure-as-Code experience
 * Terraform and/or Databricks Asset Bundles
 * Cloud experience across Azure, AWS or GCP
 * Experience delivering large-scale production data platforms
 * Strong stakeholder communication skills

For the Principal-level positions, you should also have experience setting technical direction, owning platform architecture and operating as the technical authority across large Databricks engagements.

Desirable

 * Databricks Professional-level certifications
 * Delta Sharing/Lakehouse Federation
 * Structured Streaming/Kafka
 * MLflow or AI/ML-adjacent data engineering
 * Agentic or AI-driven data engineering patterns
 * Previous consultancy or contract delivery experience

## Description

Depending on your level, you'll take ownership of designing, building and/or leading modern Databricks data platforms, working across large-scale data engineering, platform architecture, governance and deployment.

You'll work with engineering, architecture and business teams to deliver scalable production-grade solutions while helping establish strong engineering standards across the wider team., * Design and build production-grade data pipelines using Databricks, PySpark and Spark SQL

 * Design and implement Delta Lake/Medallion Architecture solutions
 * Lead technical design across ingestion, transformation and data consumption
 * Implement and optimise Unity Catalog for governance, lineage and access control
 * Build and maintain CI/CD and Infrastructure-as-Code using Terraform/Databricks Asset Bundles
 * Optimise Spark workloads for performance, scalability and cost
 * Establish engineering standards around testing, deployment and code quality
 * Work closely with architects, engineers and client stakeholders
 * Mentor and technically support other engineers
 * At Principal level, own platform-level architecture and set technical direction across Databricks workstreams


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Infrastructure as Code: The Developer's Secret Weapon](https://www.wearedevelopers.com/videos/1221-infrastructure-as-code-the-developer-s-secret-weapon)
- [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)
- [Software Engineer Salary in The UK](https://www.wearedevelopers.com/magazine/231-software-engineer-salary-in-the-uk)
- [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)