Azure Databricks Engineer

We are seeking an experienced Azure Databricks Platform Architect to design, implement, and manage scalable enterprise data platforms using Azure Databricks, Delta Lake, and modern Lakehouse architecture. The ideal candidate will have strong expertise in data engineering, platform administration, security, governance, performance optimization, and CI/CD automation within enterprise Azure environments., ·Design and implement scalable enterprise data platforms using Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage Gen2. ·Develop robust data engineering frameworks supporting batch processing, streaming, ETL/ELT pipelines, and real-time analytics. ·Administer Azure Databricks workspaces, clusters, job scheduling, compute policies, workspace governance, and secret management. ·Implement enterprise security controls, including Microsoft Entra ID, RBAC, PIM, managed identities, Key Vault, encryption, and private networking. ·Establish data governance standards using Unity Catalog for metadata management, data lineage, auditing, and fine-grained access control. ·Optimize Spark and Databricks workloads through performance tuning, partitioning, caching, Photon, cluster sizing, and cost management. ·Implement platform monitoring, logging, alerting, and operational dashboards using Azure Monitor, Log Analytics, and Databricks system tables. ·Develop reusable architecture patterns, coding standards, deployment frameworks, technical documentation, and operational procedures. ·Lead the migration of legacy data warehouses, ETL solutions, and reporting platforms to Azure Databricks and Lakehouse architecture. ·Troubleshoot production issues involving Databricks, Spark workloads, data pipelines, and Azure platform infrastructure.

·8-10 years of experience designing and implementing enterprise data platforms, including at least 5 years of hands-on Azure Databricks experience. ·Strong experience architecting modern Lakehouse solutions using Azure Databricks, Delta Lake, Unity Catalog, and ADLS Gen2. ·Proficiency in Apache Spark, PySpark, Spark SQL, Python, and advanced SQL programming. ·Strong knowledge of enterprise data governance, dimensional modeling, Data Vault, medallion architecture, and data warehouse concepts. ·Hands-on experience with Azure security, networking, monitoring, and CI/CD pipelines using Azure DevOps, automated testing, and release management., ·Technical Focus: Azure Databricks administration, Lakehouse architecture, data engineering, governance, and security. ·Security & Compliance: Experience implementing enterprise-grade access controls, encryption, auditing, and secure data access patterns. ·Operational Excellence: Ability to optimize platform performance, manage cloud costs, and support production-critical workloads. ·Documentation & Collaboration: Strong communication and collaboration skills, with the ability to work with Data Platforms, Integration, Security, and Enterprise Architecture teams.