Databricks Engineer

The Databricks Engineer is a hands-on individual contributor role within the Database Engineering team at SBA Communications, responsible for provisioning, administering, and governing the enterprise Databricks platform on Microsoft Azure. Reporting to the Manager, Database Engineering, this position requires a technically skilled professional with deep expertise in Databricks workspace administration, Unity Catalog, identity and access management, infrastructure as code, and Azure data platform integration. The Databricks Engineer will work closely with the Cloud Engineering team to ensure infrastructure-level alignment across Azure networking, storage, and security, and will serve as the primary infrastructure lead. The role plays a critical part in enforcing data governance standards, maintaining the Medallion architecture, supporting EDW/ODS platform operations, and ensuring the Databricks environment meets organizational security, compliance, and cost management requirements.

What You Will Do - Primary Responsibilities

• Databricks Platform Administration & Engineering: Provision, configure, and manage Databricks workspaces in Azure, including cluster policies, SQL Warehouses, compute governance, and workspace-level security hardening. Enforce secure cluster connectivity and no-policy cluster restrictions in alignment with EDW/ODS platform standards.
• Unity Catalog & Data Governance: Administer and maintain Databricks Unity Catalog as the system of record for data access governance, including catalog/schema structure, RBAC/ACL policies, row-level and column-level security, data lineage registration, and metadata tagging. Partner with the Manager, Database Engineering to enforce data governance standards and segregation of duties.
• Identity & Access Management: Manage Databricks user provisioning via SCIM/Automatic Identity Management integrated with Azure Active Directory/Entra ID. Maintain access control matrices, conduct quarterly access reviews (QAR), and ensure MFA enforcement and local password disablement align with organizational policy.
• Infrastructure as Code & Automation: Design and manage infrastructure provisioning using Terraform and Databricks Asset Bundles (DAB). Support CI/CD pipeline promotion across Dev, QA, and Production environments via Azure DevOps. Maintain repository structure and branching standards per SDLC guidelines.
• Medallion Architecture Support: Maintain and optimize the Bronze/Silver/Gold medallion zoning architecture within Databricks. Ensure data retention policies, naming/tagging standards, and Unity Catalog object hierarchy align with enterprise data architecture standards.
• Monitoring, Cost Management & DR: Configure and maintain monitoring using Databricks System Tables, cluster and job-level alerting, and Azure Monitor integration. Manage DBU consumption tracking, right-sizing, and cost guardrails. Support backup and disaster recovery planning in alignment with defined RTO/RPO targets.
• Collaboration with Cloud & Database Teams: Work closely with the Cloud Engineering team to ensure Databricks infrastructure integrates seamlessly with Azure networking, storage (ADLS Gen2), and security components including ExpressRoute, VNets, and firewall rules. Serve as the infrastructure handoff point to the Databricks Senior Data Engineer for pipeline and workload delivery.
• Security & Compliance: Collaborate with cybersecurity and compliance teams to enforce data platform security controls including data masking/tokenization in non-production environments, audit logging, access reviews, and SOX/segregation of duties requirements.
• Documentation & Process Improvement: Maintain accurate platform documentation and contribute to SDLC artifacts, change management templates, and runbooks for Databricks operations and incident response.
• Continuous Learning: Stay current with evolving Databricks capabilities, Azure data platform developments, Unity Catalog enhancements, and data engineering best practices. Recommend platform improvements that enhance reliability, cost-efficiency, and governance.
• Other Duties: Completes other projects and duties as assigned.

Do you have experience in Terraform?, Do you have a Bachelor's degree?, * Bachelor's Degree in Computer Science, MIS or equivalent;

• and 3-5 years of progressive experience in data platform or cloud infrastructure engineering, with hands-on expertise in Databricks or Microsoft Azure and enterprise data governance practices.

• Databricks Certified Data Engineer preferred.
• Microsoft Certified: Azure Administrator Associate (AZ-104) preferred.
• Microsoft Certified: Azure Data Engineer Associate (DP-203) preferred.
• Expert level competencies in:

• Databrick Platform Administration (workspaces, cluster policies, SQL Warehouses),
• Unity Catalog & Data Governance (RBAC, Lineage, row/column-level security)

• Advanced level competencies in:

• Azure Data Platform Integration (ALDS Gen2, Azure Networking, Entra ID/SCIM,
• Medallion Architecture (Bronze/Silver/Gold/ Delta/Lake),
• Monitoring, Cost Management & DR (System Tables, DBU tracking, RTO/RPO)
• Identity & Access Management (Entra ID, SCIM, MFA, SoD)

• Strong level competencies in:

• Infrastructure as Code (Terraform, Databricks, Asset Bundles)
• CI/CD Pipelines & DevOps (Azure DevOps, branching strategy, SDLC)
• Data Security & Compliance (data masking, audit logging, SOX controls)
• Scripting & Automation (Python, PySpark, PowerShell)

• Extensive experience with:

• Business Process Improvement
• Decision Making and Critical Thinking
• Global Perspective
• Knowledge of Organization
• Organizational Leadership
• Resource Management

Physical Demands

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

• Ability to stoop, bend, kneel or crouch.
• Ability to stand, walk and sit.
• Ability to reach with hands and arms.
• Visual ability correctable to 20/20.
• Sitting up to 90% of the day.
• Ability to respond verbally in an understandable, professional manner in person and over the telephone.
• Manual dexterity to input data into the computer and the calculator and operate the equipment listed above.
• Ability to lift up to 15 pounds.

Pulled from the full job description

• Tuition reimbursement
• Employee stock purchase plan
• 401(k) matching
• Paid time off
• Paid holidays
• RSU