Senior Data Engineer

My client is growing its technology team and building a SaaS data platform on Microsoft Azure for enterprise customers. The team is small, based in Bristol and moves fast. They need a hands-on Databricks specialist to stand up the platform, get the first pipelines live and set the patterns that future work will build on.

What you’ll be doing

• Standing up Azure Databricks as code: workspaces, Unity Catalog, permissions and environments
• Building reusable, config-driven frameworks for ingestion, medallion layers, data quality and publishing
• Delivering pipelines end to end, from source systems through to a modelled gold layer published to Azure SQL
• Implementing CDC ingestion with full history
• Setting engineering standards: CI/CD, testing, environment promotion, scheduling and alerting
• Making the platform production-ready: runbooks, backfill and replay, recovery, governance, lineage and cost control
• Supporting go-lives and stabilising pipelines after launch
• Using AI-assisted delivery with strong human review
• Coaching the engineers around you so they can run pipelines confidently without you

• 6+ years in data engineering, including 3+ years running production pipelines on Databricks
• Experience taking a Databricks platform from an empty workspace to reliable production
• Deep, current knowledge of Unity Catalog, Asset Bundles, Delta Lake (MERGE and incremental loads), PySpark, Spark SQL, Jobs and serverless compute
• Strong ingestion experience: Lakeflow Connect or equivalent CDC from SQL Server, Auto Loader, file-based sources and SCD Type 2
• A track record of building frameworks that other engineers adopted
• Proven use of AI-assisted development on production work, with good judgement about where the tools fall short
• A preference for working in person, and the ability to be in the Bristol office at the start and regularly after that
• Eligibility for UK SC clearance

Nice to have: Databricks Data Engineer Professional certification, Lakeflow Declarative Pipelines or Lakebase, Terraform, ADLS, Key Vault and Entra ID, data contracts or semantic layers, or current SC clearance. Why this role? You’ll shape how the platform is built from day one, and your work will be used long after the contract ends.