> Markdown version of [/jobs/ext/2211225-platform-architect-with-databricks](https://www.wearedevelopers.com/jobs/ext/2211225-platform-architect-with-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Platform Architect with Databricks - **Company:** Spectraforce - **Location:** United States (Remote available) - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Access Control List, Application Programming Interfaces (APIs), Microsoft Azure, Cloud Computing Security, Configuration Management, Continuous Integration, Information Engineering, Data Governance, Data Integration, Extract Transform Load (ETL), Data Security, Software Design Documents, DevOps, Disaster Recovery, Domain Name System (DNS), Github, Identity and Access Management, Virtual Private Networks (VPN), Python (Programming Language), Key Management, Network Security, Log Analysis, Routing, Performance Tuning, Windows PowerShell, Role-Based Access Control, Azure Active Directory, Azure Data Lake, Scala (Programming Language), Single Sign-On, SQL Databases, TCP/IP, Azure Service Bus, Data Logging, Scripting, Enterprise Software Applications, Azure Data Factory, Cloud Monitoring, Connectivity Problems, Apache Spark, Azure Powershell, Firewalls (Computer Science), Data Lakes, Data Lineage, Deployment Automation, Firewall Services Module, Restful APIs, Data Pipelines, Api Management, Key Vault, Databricks - **Published:** August 24, 2026 - **Apply:** http://leoforce.us/Careers/Spectraforce/JobDetails.html?jobid=74c467d7-27eb-4427-ad0a-9c8af1c65348&OrgId=1&UserId=6557 ## About the Role * Deep Expertise in Azure Services: Azure Databricks, Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, Azure Networking (VNets, NSGs, Private Endpoints), Azure Entra ID (formerly Azure AD), Azure Monitor, Log Analytics. * Databricks Specifics: Unity Catalog, Delta Lake, Spark architecture, Databricks Runtime, Workspaces, Clusters, Jobs, Notebooks, Repos, Databricks APIs. * Security: Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security. Experience with SCIM. * Networking: Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure. * Scripting/Automation: Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks. * CI/CD: Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions). * Problem-Solving: Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues. * Communication: Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences. * Certifications (Preferred): Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Professional/Associate. ## Description * Design & Deployment: Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints. * Configuration Management: Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations. * Performance Tuning: Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations. * Capacity Planning: Forecast and plan for future Databricks resource needs based on platform usage trends. * Cost Optimization: Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances). * Troubleshooting: Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems. 2. Data Governance & Unity Catalog: * Unity Catalog Design: Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations. * Fine-Grained Access Control (FGAC): Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required. * Data Lineage & Audit: Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access. * Managed Tables & Volumes: Advise and govern the usage of managed and external tables/volumes within Unity Catalog. 3. Security & Access Management (Entra ID, SCIM, Key Vault, ACLs): * Azure Entra ID Integration: Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization. * SCIM (System for Cross-domain Identity Management): Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks. * Azure Key Vault Integration: Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations. * Access Control Lists (ACLs): Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments. * Compliance & Auditing: Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity. 4. Networking & Connectivity: * Secure Network Design: Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs). * Firewall Rules: Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources. * DNS & Routing: Ensure proper DNS resolution and routing for Databricks components and integrated services. 5. Data Integration & Orchestration (Azure Data Factory - ADF): * ADF Integration Patterns: Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines. * Automated Deployments: Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps. * API Management: Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed. 6. Azure Storage (ADLS Gen2) Integration: * Storage Account Design: Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies. * Mount Points & Credential Passthrough: Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals. 7. Operations, Monitoring & Reliability: * Monitoring & Alerting: Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization). * Disaster Recovery & Business Continuity: Contribute to and validate disaster recovery and business continuity plans for the Databricks platform. * Patching & Upgrades: Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities. * Service Health: Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks. 8. Documentation & Best Practices: * Architectural Documentation: Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform. * Best Practices: Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams. * Knowledge Transfer: Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage. 9. Collaboration & Stakeholder Management: * Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues. * Act as a subject matter expert for all Databricks-related queries from internal teams. * Participate in architectural review boards and provide technical leadership. ## Related Videos - [Automated MS SQL Server database deployments with dacpacs and Azure DevOps](https://www.wearedevelopers.com/videos/334-automated-ms-sql-server-database-deployments-with-dacpacs-and-azure-devops) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [An Applied Introduction to eBPF with Go](https://www.wearedevelopers.com/videos/1075-an-applied-introduction-to-ebpf-with-go) - [DevOps Maturity Check – a way to balance autonomy and alignment](https://www.wearedevelopers.com/videos/58-devops-maturity-check-a-way-to-balance-autonomy-and-alignment) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)