SC Cleared Databricks Engineer

The successful candidates will play a key role in designing, developing, and optimising data pipelines and analytics solutions using Databricks within a secure environment. You will work with large and complex datasets from UK and potentially international sources, enabling advanced analytics and data-driven insights to inform strategic decisions.

This role is highly technical and data-focused, requiring strong experience in Databricks, Spark, and related data engineering tools. While stakeholder engagement is minimal, you will need to align technical solutions with business objectives and ensure data integrity and compliance throughout.

Key Requirements

• Active SC Clearance (used within the last 12 months)
• Proven experience with Databricks (including notebooks, clusters, and job orchestration)
• Strong knowledge of Apache Spark, PySpark, and distributed data processing
• Experience building and optimising ETL pipelines and data workflows
• Familiarity with Delta Lake, SQL, and data modelling best practices
• Ability to work with large, complex datasets from multiple sources
• Comfortable working independently in a fully remote environment
• Strong understanding of Agile delivery (Scrum/Kanban)
• Excellent attention to detail and structured approach to analysis
• Ability to interpret business needs and translate them into scalable data solutions

Desirable

• Experience with Azure Databricks and related cloud services (Azure Data Lake, Data Factory)
• Knowledge of data governance, security, and compliance in government environments
• Familiarity with CI/CD pipelines for data engineering