Databricks Data Engineer - Azure Cloud & Performance Optimization
Brightbox Grp Ltd
Charing Cross, United Kingdom
8 days ago
Role details
Contract type
Permanent contract Employment type
Part-time (≤ 32 hours) Working hours
Regular working hours Languages
English Compensation
£ 119KJob location
Remote
Charing Cross, United Kingdom
Tech stack
Azure
Continuous Integration
ETL
Data Transformation
Data Systems
Document Management Systems
Data Intelligence
Performance Tuning
Power BI
Azure
Spark
Data Lake
PySpark
Azure
Data Pipelines
Serverless Computing
Databricks
Job description
We are seeking a Databricks Data Engineer experienced in building high-quality data pipelines and optimizing large-scale workloads on the Databricks Data Intelligence Platform. The ideal candidate is strong in PySpark, Delta Lake, Azure services, workflow automation, and performance tuning., * Build and maintain scalable batch and streaming pipelines using Databricks notebooks, Jobs, and Workflows.
- Optimize Spark and Delta Lake performance through cluster tuning, efficient configurations, AQE, and caching.
- Run performance tests and ensure workloads are cost-efficient and reliable.
- Apply data quality, lineage, and access controls using Unity Catalog and governance best practices.
- Develop reusable PySpark ETL and data transformation code.
- Manage Delta Lake tables with ACID transactions, schema evolution, and time-travel capabilities.
- Integrate Databricks with Azure ADLS, Key Vault, Azure Functions, and related services.
- Work with architects and analysts to design end-to-end data solutions for analytics and ML.
- Support CI/CD automation for Databricks via Azure DevOps or similar tools.
- Maintain documentation on architecture, performance, and governance settings.
Requirements
- Strong experience with Databricks and multi-cloud data ecosystems.
- Hands-on with Databricks Jobs/Workflows and monitoring.
- Advanced PySpark skills, including optimization and schema management.
- Deep understanding of Delta Lake and incremental processing.
- Proven Spark tuning and cluster optimization expertise.
- Experience with Unity Catalog security, lineage, and access control.
- Practical knowledge of Azure ADLS Gen2, Key Vault, Azure Functions.
- Familiarity with CI/CD automation for Databricks.
- Strong analytical and troubleshooting abilities., * Experience with large enterprise Databricks environments and governed catalogs.
- Knowledge of Azure Synapse, Power BI, or similar analytics tools.
- Understanding of Databricks cost-optimization strategies.
- Strong communication and cross-team collaboration skills.
