> Markdown version of [/jobs/ext/3553677-databricks-technical-lead-architect](https://www.wearedevelopers.com/jobs/ext/3553677-databricks-technical-lead-architect). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Technical Lead/Architect - **Company:** VISION INFOTECH INC. - **Location:** Melrose, United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Query Performance, Application Programming Interfaces (APIs), Artificial Intelligence, Amazon Web Services, Business Analytics Applications, Application Integration Architecture, Business Logic, Computing Platforms, Architectural Patterns, Automation of Tests, Microsoft Azure, Big Data, Cloud Computing, Software Quality, Code Review, Databases, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Data Security, Data Sharing, Software Debugging, Software Design Patterns, DevOps, Distributed Data Store, Identity and Access Management, Python (Programming Language), Machine Learning, Performance Tuning, Query Optimization, Power BI, Azure Active Directory, Cloud Services, Data Mesh, Standard Sql, Azure Machine Learning, SQL Databases, Data Streaming, Tableau (Software), Azure Service Bus, Feature Store, Google Cloud, Data Classification, Delivery Pipeline, Multi-Cloud, Change Data Capture, Git, Cloudformation, Event Driven Architecture, Data Lakes, Pyspark, Debezium, Data Lineage, Qlikview, Apache Kafka, Data Management, Machine Learning Operations, Software Coding, Terraform, Domain Driven Design, Azure Synapse Analytics, Looker Analytics, Software Version Control, Data Pipelines, Serverless Computing, Azure Resource Manager, Databricks - **Published:** October 2, 2026 - **Apply:** https://www.dice.com/job-detail/5f6ab2a3-ba34-442f-8821-89bd3f3a9652 ## About the Role · Current, hands-on development experience is essential. This role requires the ability to read, write, debug, and lead code reviews of production PySpark and SQL code-not just architectural oversight. Candidates must demonstrate recent (within past 12 months) hands-on development work including debugging live code, explaining business logic and technical implementation, optimizing queries, and implementing data pipelines. Ability to comprehend and explain unfamiliar code samples is a core requirement. · Advanced proficiency in SQL and Python/PySpark with demonstrated ability to write complex transformations, optimize query performance, explain code logic at both business and technical levels, and troubleshoot production issues. Must be comfortable with CTEs, window functions, table-valued functions, APPLY operators, advanced SQL operators, and PySpark DataFrame/RDD operations. Current, active coding skills required-not aspirational or theoretical knowledge. · 7+ years of experience in data architecture, data engineering, or platform engineering roles, with at least 3+ years focused on Databricks platform architecture. · Expert-level knowledge of Databricks platform components: Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL Warehouses, MLflow, and Databricks SQL. · Deep expertise in Unity Catalog governance, including metastore design, catalog/schema strategies, permission models, data lineage, and multi-workspace/multi-cloud patterns. · Strong architectural background in cloud platforms (Azure, AWS, or Google Cloud Platform), including storage services, identity management (Azure AD, AWS IAM), networking, and security best practices. · Proven experience designing enterprise-scale data architectures, including medallion/multi-hop architectures, data mesh patterns, domain-driven design, and data product frameworks. · Hands-on experience with infrastructure-as-code (Terraform, ARM templates, CloudFormation) for platform configuration and governance automation. · Strong understanding of DevOps practices, CI/CD pipelines, version control strategies, and automated testing for data platforms. · Experience with performance tuning, cost optimization, and capacity planning for large-scale data platforms. Preferred Qualifications · Databricks certification (e.g., Databricks Certified Data Engineer Professional, Solutions Architect). Note: Certification alone is insufficient; candidates must demonstrate current hands-on technical execution aligned with certification level. · Cloud certifications such as Azure Solutions Architect, AWS Solutions Architect, or Google Cloud Platform Professional Data Engineer. * Experience designing multi-tenant architectures with secure data isolation, cross-tenant data sharing, and compliance controls. * Background in streaming architectures using Structured Streaming, Kafka, Event Hubs, or Kinesis. * Exposure to machine learning operations (MLOps), feature stores, model serving, and AI/ML platform architecture. * Experience with data mesh implementations, federated governance, and distributed data ownership models. * Knowledge of analytics platforms (Power BI, Tableau, Looker) and their integration patterns with Databricks. * Familiarity with domain-specific data models in education (CEDS), healthcare, finance, or operational domains. * Experience with real-time CDC patterns, change data capture tools (Debezium, Qlik, Fivetran), and event-driven architectures. Leadership & Soft Skills · Professional presence and composure under pressure, including ability to handle unexpected technical challenges, maintain calm in production incidents, and communicate effectively during high-stress situations. · Strategic thinking with ability to balance long-term architectural vision with pragmatic, incremental delivery. · Exceptional communication skills to articulate complex architectural concepts to technical and non-technical stakeholders, including executive leadership. · Proven ability to influence and drive consensus across multiple teams and organizational levels. · Mentorship and enablement mindset to uplift engineering teams through knowledge sharing, documentation, and hands-on guidance. · Strong problem-solving capabilities with a focus on root cause analysis and sustainable solutions. · Commitment to quality, including comprehensive documentation, architectural decision records (ADRs), and knowledge transfer. ## Description · Technical Leadership: Establish architectural patterns and best practices for medallion architecture, Delta Lake optimization, data mesh principles, pipeline orchestration, and cross-domain data sharing. Lead technical code reviews for data engineering teams, providing detailed feedback on PySpark, SQL, and architectural patterns; ensure code quality, performance optimization, and adherence to best practices. · Architecture & Strategy: Define the overall Databricks platform architecture, including workspace design, Unity Catalog governance model, storage patterns, compute strategies, and integration with broader enterprise data ecosystems. · DevOps & Automation: Design CI/CD architecture for Databricks assets, including Git-based development workflows, automated testing frameworks, deployment pipelines, and infrastructure-as-code patterns using Terraform or similar tools. · Governance & Security: Design and implement comprehensive governance frameworks within Unity Catalog, including multi-catalog strategies, metastore configuration, identity federation, fine-grained access controls, attribute-based access control (ABAC), data classification, and compliance requirements. · Platform Design: Architect solutions for compute resource allocation (all-purpose clusters, job clusters, SQL warehouses, serverless), autoscaling strategies, cost optimization, and workload isolation across teams and environments. · Data Modeling & Standards: Define enterprise data modeling standards, schema design patterns, slowly changing dimensions (SCD) strategies, CDC architectures, and data quality frameworks that scale across multiple domains. · Performance & Optimization: Architect solutions for query optimization, storage layout strategies (partitioning, Z-ordering, liquid clustering), caching, materialized views, and monitoring/observability across the platform. · Integration Architecture: Design integration patterns with upstream source systems (APIs, databases, streaming), downstream analytics tools (Power BI, Tableau), and adjacent cloud services (Azure Synapse, AWS services, etc.). · Collaboration & Enablement: Partner with engineering teams, data teams, and stakeholders to translate business requirements into architectural blueprints; mentor engineers on implementation of architectural patterns. · Innovation & Evaluation: Stay current with Databricks platform evolution, evaluate new features (Lakehouse Federation, AI/ML capabilities, streaming enhancements), and drive adoption of capabilities that deliver business value., * This role is ideal for a seasoned architect who thrives on designing elegant, scalable solutions and wants to shape the foundation of a next-generation Lakehouse platform. The Databricks Architect will serve as the technical authority for the platform, driving architectural excellence, governance maturity, and innovation that enables the organization to unlock the full value of its data assets across analytics, data science, and operational use cases. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Shifting Stress to Progress— Understanding DevOps to do DevOps Better](https://www.wearedevelopers.com/videos/268-shifting-stress-to-progress-understanding-devops-to-do-devops-better) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Fully Orchestrating Databricks from Airflow](https://www.wearedevelopers.com/videos/336-fully-orchestrating-databricks-from-airflow) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Résumé-Driven Development: How IT trends affect the job market for software developers](https://www.wearedevelopers.com/magazine/59-resume-driven-development-how-it-trends-affect-the-job-market-for-software-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)