Databricks Engineer - Remote

• Design, develop, and maintain ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake.
• Build and optimize data lakehouse architectures on Azure Data Lake Storage (ADLS).
• Develop high-performance data solutions using Azure Synapse, Azure Data Factory, and Databricks workflows.
• Implement best practices for data governance, security, and quality across all pipelines.
• Collaborate with data scientists, analysts, and cross-functional teams to deliver reliable, production-grade data models.
• Monitor and tune pipeline performance to ensure efficiency, reliability, and cost optimization.
• Participate in CI/CD processes and infrastructure-as-code solutions using tools like Terraform, GitHub Actions, or Azure DevOps.

Technologies:

• AI
• Azure
• CI/CD
• Cloud
• Databricks
• DevOps
• ETL
• GitHub
• Support
• Kafka
• Python
• PySpark
• SQL
• Security
• Spark
• Terraform
• Unity
• GameDev

• Over 3 years of experience as a Data Engineer in Azure environments.
• Strong hands-on experience with Databricks, including PySpark, Delta Lake, cluster optimization, and job scheduling.
• Solid knowledge of Azure cloud services, including:
• • Azure Data Lake Storage
• • Azure Data Factory
• • Azure Synapse / SQL Pools
• • Azure Key Vault
• Proficient programming skills in Python and SQL.
• Experience building scalable, production-grade data pipelines.
• Understanding of data modeling, data warehousing concepts, and distributed computing.
• Familiarity with CI/CD, version control, and DevOps practices., We are excited to welcome a highly skilled Azure Data Engineer with deep, hands-on Databricks experience to our growing data engineering team. In this role, you will play a crucial part in designing, building, and optimizing scalable data pipelines and lakehouse architectures on Azure, ultimately enabling advanced analytics and data-driven decision-making throughout the business.

Additionally, while not mandatory, it would be great if you have experience with streaming technologies (e.g., Spark Structured Streaming, Event Hub, Kafka), knowledge of MLflow, Unity Catalog, or advanced Databricks features, exposure to Terraform or other Infrastructure as Code (IaC) tools, and experience working in Agile/Scrum environments.

We pride ourselves on being a leading recruiter for Data & AI roles in the UK, offering more opportunities across the country than any other recruitment agency. We also proudly sponsor and support events like SQLBits, Power Platform World Tour, and the London Fabric User Group. Join us and elevate your career in Data & AI recruitment.