Senior Azure Databricks Data Engineer - Banking Client

We are looking for an experienced Senior Azure Databricks Data Engineer to join a large-scale enterprise data transformation programme within a complex financial services environment.

This is a hands-on engineering position for someone who enjoys designing, developing and optimising production-grade data solutions rather than operating purely at architecture or management level.

You will work within a modern Microsoft Azure and Databricks ecosystem, building scalable data pipelines, Lakehouse solutions and reusable engineering components that support analytics, applications and emerging AI/ML use cases.

The role combines Data Engineering, Databricks/Spark development and software engineering, so we are particularly interested in engineers with strong coding skills and experience taking data solutions from design through to production.

What You’ll Be Doing:

Azure Databricks & Data Engineering

• Design, build, test and maintain scalable data engineering solutions using Microsoft Azure and Azure Databricks.
• Develop production-grade ETL/ELT pipelines using Python, PySpark, Scala and SQL.
• Build data-processing applications covering ingestion, transformation, enrichment, validation and serving.
• Develop Delta Lake / Lakehouse solutions using modern data engineering patterns.
• Design and implement Bronze, Silver and Gold / Medallion architectures where appropriate.
• Build reliable batch and streaming data-processing workflows.
• Develop curated datasets and transformation layers supporting analytics, reporting, applications and AI/ML use cases.

Databricks & Apache Spark

• Develop and maintain Databricks notebooks, jobs and workflows.
• Build and optimise Apache Spark / PySpark workloads operating across large datasets.
• Improve Spark performance through appropriate partitioning, caching, cluster configuration and query optimisation.
• Implement schema evolution, incremental processing and robust data-quality controls.
• Develop reusable Spark/Python components, libraries and engineering frameworks.
• Apply appropriate Delta Lake optimisation and data-management techniques.
• Troubleshoot and optimise production data pipelines for performance, scalability, reliability and cost.

Azure Data Platform

Work across a modern Azure data ecosystem including:

• Azure Databricks
• Apache Spark / PySpark
• Delta Lake
• Azure Data Lake Storage (ADLS)
• Azure Data Factory
• Azure Synapse
• Azure Event Hubs / streaming patterns
• Azure Key Vault
• Azure DevOps
• Azure monitoring and logging capabilities

You will work closely with Cloud, Architecture and Platform teams to ensure solutions are secure, scalable, observable and aligned with enterprise standards.

Software Engineering & Application Development

This role goes beyond traditional ETL development.

You will apply strong software engineering practices to data applications, including:

• Modular and reusable development
• Clean, maintainable code
• Automated testing and validation
• Error handling and logging
• Code reviews
• Git/version control
• Technical documentation
• Reusable libraries and frameworks

You will be expected to contribute to the overall quality of the engineering environment rather than simply delivering individual pipelines.

DevOps & CI/CD

• Build and maintain CI/CD processes for data applications.
• Deploy Databricks and data-engineering code across development, test and production environments.
• Work with Azure DevOps and YAML pipelines.
• Collaborate with DevOps and Cloud teams on environment configuration and deployment.
• Apply release-management and environment-promotion best practices.
• Work with Terraform / Infrastructure as Code where required.

Terraform expertise is beneficial, but this is primarily a Data Engineering and application-development role rather than an Infrastructure Engineering position.

Data Quality, Security & Governance

• Build data-quality controls and validation into engineering pipelines.
• Implement appropriate logging, monitoring and operational alerting.
• Work with enterprise security and access-management standards.
• Apply secure coding and cloud data-engineering practices.
• Support metadata, lineage and governance requirements.
• Consider performance and cloud cost when designing and developing solutions.

Technical Leadership

As a senior member of the engineering team, you will also:

• Work closely with Data, Architecture, AI, Cloud, DevOps and application teams.
• Translate complex business and analytical requirements into practical engineering solutions.
• Contribute to technical design and engineering standards.
• Conduct code reviews and provide technical guidance.
• Support and mentor less experienced engineers.
• Help establish reusable development patterns and engineering best practices.

You should have at least 5 years of hands-on Data Engineering / Data Application Development experience, ideally within large enterprise environments.

Core Technical Skills

Strong hands-on experience with:

• Azure Databricks
• Python
• PySpark
• Apache Spark
• Scala
• SQL
• ETL / ELT pipeline development
• Delta Lake / Lakehouse architecture
• Azure Data Factory
• Azure Data Lake Storage
• Azure DevOps / CI/CD

We are particularly interested in candidates who can demonstrate that they have personally designed and developed production Databricks/PySpark solutions, rather than only managing teams or defining architecture.

Highly Desirable

Experience with any of the following would be advantageous:

• Spark performance optimisation
• Databricks Auto Loader
• Delta Live Tables
• Unity Catalog
• Medallion / Bronze-Silver-Gold architecture
• Streaming data pipelines
• Azure Event Hubs
• Kafka
• Azure Synapse
• Azure Key Vault
• Azure DevOps YAML
• Terraform
• Data quality frameworks
• Metadata management and lineage
• MLflow / MLOps
• AI/ML data pipelines
• Reusable data-engineering frameworks or libraries
• Enterprise security and governance

Certifications

Relevant certifications are advantageous, particularly:

• Microsoft Azure Data Engineer
• Microsoft Azure Developer
• Microsoft Azure Solutions Architect
• Microsoft Azure DevOps Engineer
• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Databricks Certified Developer for Apache Spark

The Profile That Will Stand Out

The strongest candidate will be someone who can say:

“I personally build production Azure Databricks solutions. I write Python/PySpark code, build and optimise Spark pipelines, work with Delta Lake and ADF, automate deployments through CI/CD, troubleshoot production issues and understand how to engineer scalable data solutions rather than simply design them.”

This is not primarily a BI, reporting or high-level Data Architecture position. We are looking for a genuinely hands-on senior engineer who is comfortable getting into the code.