Databricks Engineer (Security Clearance Active)

• Build and optimise Databricks notebooks, pipelines, and Delta Lake workflows.
• Develop scalable ETL/ELT processes for large, secure datasets.
• Implement best practices for data governance, quality, and access control.
• Work with architects and security teams to design secure data solutions.
• Troubleshoot performance issues and contribute to platform enhancements.
• Deliver high-quality engineering outputs within agile delivery cycles.

• Current SC or DV clearance (mandatory).
• Strong commercial experience with Azure Databricks.
• Advanced PySpark or Scala Spark engineering skills.
• Experience delivering data solutions in secure, regulated, or government environments.
• Hands-on expertise with Azure Data Lake, Data Factory, Key Vault, DevOps Pipelines.
• Strong understanding of Delta Lake, distributed processing, and performance optimisation.

Nice to Have

• Experience with Unity Catalog, MLflow, or CI/CD automation for Databricks.
• Familiarity with Terraform or IaC for secure environments.
• Knowledge of data governance frameworks used across UK Government.