Platform Architect with Databricks

Data Governance & Unity Catalog: * Unity Catalog Design: Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations. * Fine-Grained Access Control (FGAC): Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required. * Data Lineage & Audit: Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access. * Managed Tables & Volumes: Advise and govern the usage of managed and external tables/volumes within Unity Catalog.

Security & Access Management (Entra ID, SCIM, Key Vault, ACLs): * Azure Entra ID Integration: Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization. * SCIM (System for Cross-domain Identity Management): Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks. * Azure Key Vault Integration: Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations. * Access Control Lists (ACLs): Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments. * Compliance & Auditing: Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity.

Networking & Connectivity: * Secure Network Design: Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs). * Firewall Rules: Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources. * DNS & Routing: Ensure proper DNS resolution and routing for Databricks components and integrated services.

Data Integration & Orchestration (Azure Data Factory - ADF): * ADF Integration Patterns: Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines. * Automated Deployments: Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps. * API Management: Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed.

Azure Storage (ADLS Gen2) Integration: * Storage Account Design: Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies. * Mount Points & Credential Passthrough: Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals.

Operations, Monitoring & Reliability: * Monitoring & Alerting: Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization). * Disaster Recovery & Business Continuity: Contribute to and validate disaster recovery and business continuity plans for the Databricks platform. * Patching & Upgrades: Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities. * Service Health: Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks.

Documentation & Best Practices: * Architectural Documentation: Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform. * Best Practices: Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams. * Knowledge Transfer: Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage.

Collaboration & Stakeholder Management: * Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues. * Act as a subject matter expert for all Databricks-related queries from internal teams. * Participate in architectural review boards and provide technical leadership.