Azure Databricks Engineer

• Design, develop, and maintain scalable and reliable data processing solutions using Azure Databricks.

• Build and manage robust streaming and batch data pipelines within complex Databricks environments.

• Design and optimize data models to support scalable processing and performance requirements.

• Manage multiple parallel data processing workflows using shared source datasets.

• Develop and maintain data transformation logic using Python, PySpark, and SQL.

• Implement and manage CI/CD pipelines using Azure and YAML-based configurations.

• Utilize Infrastructure as Code (ARM/Bicep) for deployment and infrastructure automation.

• Collaborate with cross-functional teams to ensure high-quality, maintainable, and reliable solutions.

• Continuously improve system stability, performance, and engineering practices while minimizing technical debt.

• Contribute actively within Agile teams to deliver scalable and sustainable data solutions.

• 8-10 years of overall professional experience, with strong expertise in Azure Databricks and Data Engineering.

• Deep hands-on experience with Azure Databricks as the primary technology platform.

• Strong proficiency in Python, PySpark, and SQL.

• Proven experience building and managing streaming and batch data processing solutions.

• Strong experience designing and maintaining scalable Databricks data models and architectures.

• Solid understanding of Azure cloud services and YAML-based CI/CD pipelines.

• Hands-on experience with Infrastructure as Code (ARM/Bicep).

• Experience working within Agile methodologies and delivery environments.

• Broader understanding of modern end-to-end data solution architectures is an advantage.

• Strong communication and collaboration skills.

You Should Possess the Ability to:

• Build scalable, high-performance, and reliable data processing solutions.

• Design pragmatic and maintainable architectures while avoiding unnecessary complexity.

• Manage multiple data workloads and parallel processing scenarios efficiently.

• Troubleshoot, optimize, and improve complex data environments.

• Make well-informed technical decisions in evolving architectural landscapes.

• Collaborate effectively across engineering and business teams.

• Drive continuous improvement with a strong quality-first mindset.