Expert Lead Azure Databricks Engineer

We are seeking an experienced and highly skilled Lead Azure Databricks Engineer to join our dynamic team. The successful candidate will be responsible for designing, developing, and leading complex data engineering solutions within the Azure cloud environment. This role offers an exciting opportunity to work on cutting-edge data projects, optimise data workflows, and mentor junior team members. The ideal applicant will possess extensive expertise in Azure Databricks, big data processing, and cloud architecture, ensuring the delivery of scalable and efficient data solutions that support strategic business objectives., * Lead the design and implementation of large-scale data pipelines using Azure Databricks, ensuring optimal performance and scalability.

• Collaborate with cross-functional teams including data scientists, analysts, and software engineers to develop integrated data solutions.
• Architect and optimise cloud-based data workflows, ensuring best practices in security, compliance, and cost-efficiency.
• Provide technical leadership and mentorship to junior engineers, fostering a culture of continuous improvement and innovation.
• Develop comprehensive documentation for data processes, architecture diagrams, and operational procedures.
• Troubleshoot complex data issues promptly and implement robust solutions to prevent recurrence.
• Stay abreast of emerging technologies in big data processing and cloud computing to recommend enhancements to existing systems.
• Participate in project planning, estimation, and stakeholder communication to ensure timely delivery of deliverables.

• Proven experience as a Lead Data Engineer or similar role specialising in Azure Databricks within a cloud environment.
• Extensive knowledge of Spark architecture, PySpark/Scala programming languages, and big data frameworks.
• Strong understanding of Azure services such as Data Factory, Data Lake Storage, Synapse Analytics, and related cloud security protocols.
• Demonstrated ability to design scalable ETL/ELT pipelines with high performance optimisation techniques.
• Experience with CI/CD pipelines for automated deployment of data solutions.
• Excellent problem-solving skills with a proactive approach to troubleshooting complex technical issues.
• Strong communication skills with the ability to convey technical concepts clearly to non-technical stakeholders.
• Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or equivalent are highly desirable. This role offers an engaging environment where innovation is encouraged, and your expertise will directly impact the organisation's success in harnessing the power of big data analytics within the Azure ecosystem.

Pay: £400.00-£500.00 per day