Databricks Architect

· 12 YRS Exp

· Design and implement enterprise-scale solutions on the Databricks Lakehouse Platform.

· Architect end-to-end data pipelines for batch and real-time processing using Apache Spark and PySpark.

· Develop scalable data ingestion, transformation, and data quality frameworks.

· Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.

· Build and optimize data warehouses, data marts, and analytical solutions.

· Implement data governance, security, lineage, and access controls using Unity Catalog.

· Develop and support AI/BI dashboards, semantic models, and self-service analytics solutions.

· Configure and optimize Genie Spaces to enable natural language business queries and conversational analytics.

· Design and deploy Generative AI and RAG-based solutions using Databricks Mosaic AI and Vector Search.

· Collaborate with business users to translate requirements into scalable data and AI solutions.

· Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.

· Lead cloud-native implementations across Azure environments.

· Define architecture standards, best practices, and reusable design patterns.

· Mentor data engineers, analysts, and architects on Databricks technologies and platform adoption.

· Lead migration initiatives from legacy data warehouses and analytics platforms to Databricks.

· Build and maintain Genie Spaces for business self-service analytics.

· Create semantic models, metrics, and trusted data assets for AI-driven reporting.

· Develop natural language-to-SQL analytics solutions using Databricks Genie.

· Implement RAG solutions using enterprise data and Vector Search.

· Optimize AI/BI dashboards and conversational analytics experiences.

· Troubleshoot Spark performance, query optimization, and workload management.

· Automate data validation, monitoring, and governance controls.

· Support AI use cases using Mosaic AI model serving and inference endpoints.

· Databricks Lakehouse Platform

· Apache Spark, PySpark, Spark SQL

· Python, SQL

· Delta Lake, Delta Live Tables, Lakeflow

· Unity Catalog

· Databricks AI/BI and Genie

· Mosaic AI, Vector Search, RAG

· Data Modeling (Dimensional & Data Vault)

· Structured Streaming

· Data Quality and Data Governance

· Azure

· Terraform, Git, Azure DevOps, Jenkins

· REST APIs and Data Integration

· Performance Tuning and Cost Optimization

Certification : Azure Databricks certified Data Eng professional