Azure Databricks Engineer

We are seeking an experienced Senior Databricks Engineer with deep expertise in Azure cloud data engineering, large-scale data processing, and enterprise analytics. The ideal candidate will design, develop, optimize, and maintain scalable data platforms handling approximately 50 TB of data, 10,000+ tables, and 300 dashboards., * Design, build, and maintain scalable data engineering solutions using Azure Databricks and Microsoft Azure services.

• Develop and optimize ETL/ELT pipelines to process and manage approximately 50 TB of enterprise data.
• Manage and optimize data architecture involving 10,000+ tables across multiple source systems.
• Support, enhance, and optimize approximately 300 business intelligence dashboards and their underlying data pipelines.
• Develop robust data ingestion, transformation, aggregation, and data quality frameworks using PySpark, Spark SQL, and Python.
• Implement lakehouse architecture using Delta Lake, Delta tables, and scalable data storage patterns.
• Optimize Spark jobs, cluster configurations, partitioning, caching, and query execution to improve performance and control costs.
• Integrate Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, and other relevant Azure services with Databricks.
• Architect integration patterns with cloud services (e.g., Azure ADLS Gen2 / AWS S3, Azure Synapse / Snowflake, Kafka, Event Hubs, Power BI / Tableau).
• Build reliable data models and curated datasets to support Power BI and other enterprise reporting platforms.
• Implement data governance, access controls, lineage, monitoring, and security using Unity Catalog and applicable Azure security services.
• Troubleshoot complex production issues, data pipeline failures, performance bottlenecks, and data inconsistencies.
• Design and implement AI/LLM-powered data solutions using Claude, Vertex AI, or comparable AI platforms, where applicable.
• Develop AI-assisted data discovery, metadata search, natural-language-to-SQL, and analytics automation capabilities.
• Collaborate with data architects, BI developers, analytics teams, AI engineers, and business stakeholders across distributed teams.
• Establish best practices for CI/CD, version control, automated testing, deployment, and production support.

The candidate should have strong experience with Azure Databricks, Delta Lake, Apache Spark, data pipeline architecture, performance tuning, and enterprise-scale data governance. Experience integrating AI capabilities using Claude (Anthropic), Google Vertex AI, or similar large language model (LLM) platforms is highly desirable., Azure & Databricks

• Azure Databricks and Databricks Lakehouse Platform
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS)
• Delta Lake and Delta Live Tables / Lakeflow Declarative Pipelines
• Unity Catalog and data governance

Data Engineering

• Apache Spark, PySpark, Spark SQL, and Python
• Advanced SQL, query optimization, and data modeling
• ETL/ELT pipeline development and orchestration
• Large-scale data processing and distributed computing
• Data quality, reconciliation, and pipeline monitoring

Enterprise Data Scale

• Experience handling multi-terabyte datasets, ideally around 50 TB or more
• Experience managing complex data estates with thousands of tables
• Experience supporting large BI environments with hundreds of dashboards
• Performance optimization, workload management, scalability, and cost optimization

AI & LLM Technologies

• Experience integrating LLMs into data engineering or analytics workflows
• Familiarity with Anthropic Claude APIs and/or Google Vertex AI
• Prompt engineering, API integration, and secure handling of AI requests
• Experience with RAG, vector search, metadata-driven data discovery, or natural-language analytics is a plus

DevOps & Security

• Git, CI/CD pipelines, and automated deployments
• Azure DevOps or GitHub Actions
• Identity and access management, secrets management, and secure data access
• Production monitoring, logging, and incident resolution

Preferred Qualifications

• Experience migrating legacy data warehouses or on-premises platforms to Azure Databricks.
• Experience with Power BI semantic models, datasets, refresh pipelines, and dashboard performance.
• Experience with Microsoft Purview or comparable data catalog and governance platforms.
• Familiarity with enterprise data architecture, dimensional modeling, and medallion architecture.
• Experience working with regulated or security-sensitive enterprise data.
• Relevant Azure or Databricks certifications.

Success Criteria

• Reliable processing and transformation of approximately 50 TB of data.
• Effective management and optimization of a data environment containing 10,000+ tables.
• Stable, performant data pipelines supporting approximately 300 dashboards.
• Improved data quality, pipeline reliability, query performance, and cloud cost efficiency.
• Secure and scalable integration of AI/LLM capabilities into enterprise data workflows.