> Markdown version of [/jobs/ext/4396074-azure-databricks-engineer](https://www.wearedevelopers.com/jobs/ext/4396074-azure-databricks-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Azure Databricks Engineer - **Company:** Cal's Auto Service Inc - **Location:** United States (Remote available) - **Experience:** Expert - **Salary:** $145,600.0 - $166,400.0 - **Contract:** Temporary contract - **Skills:** Query Performance, Application Programming Interfaces (APIs), Artificial Intelligence, Amazon S3, Automation of Tests, Microsoft Azure, Business Intelligence Development, Big Data, Continuous Integration, Data Architecture, Data Discovery, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Security, Data Systems, Data Warehousing, DevOps, Dimensional Modeling, Distributed Systems, Github, Apache Hive, Identity and Access Management, Python (Programming Language), Key Management, Metadata, Performance Tuning, Query Optimization, Power BI, Cloud Services, Azure Data Lake, Search Technologies, SQL Databases, Systems Integration, Tableau (Software), Enterprise Data Management, Azure Service Bus, Data Logging, Data Storage Technologies, Data Ingestion, Azure Data Factory, Sql Optimization, Retrieval-Augmented Generation, Large Language Models, Snowflake, Prompt Engineering, Apache Spark, Google Vertex AI, Caching, Git, Data Lakes, AI Platforms, Pyspark, Deployment Automation, Apache Kafka, Data Management, Claude, Cloud Optimization, Azure Synapse Analytics, Data Inconsistencies, Software Version Control, Data Pipelines, Databricks - **Published:** October 11, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=2ef0def19f3a7249 ## About the Role The candidate should have strong experience with Azure Databricks, Delta Lake, Apache Spark, data pipeline architecture, performance tuning, and enterprise-scale data governance. Experience integrating AI capabilities using Claude (Anthropic), Google Vertex AI, or similar large language model (LLM) platforms is highly desirable., Azure & Databricks * Azure Databricks and Databricks Lakehouse Platform * Azure Data Factory (ADF) * Azure Data Lake Storage Gen2 (ADLS) * Delta Lake and Delta Live Tables / Lakeflow Declarative Pipelines * Unity Catalog and data governance Data Engineering * Apache Spark, PySpark, Spark SQL, and Python * Advanced SQL, query optimization, and data modeling * ETL/ELT pipeline development and orchestration * Large-scale data processing and distributed computing * Data quality, reconciliation, and pipeline monitoring Enterprise Data Scale * Experience handling multi-terabyte datasets, ideally around 50 TB or more * Experience managing complex data estates with thousands of tables * Experience supporting large BI environments with hundreds of dashboards * Performance optimization, workload management, scalability, and cost optimization AI & LLM Technologies * Experience integrating LLMs into data engineering or analytics workflows * Familiarity with Anthropic Claude APIs and/or Google Vertex AI * Prompt engineering, API integration, and secure handling of AI requests * Experience with RAG, vector search, metadata-driven data discovery, or natural-language analytics is a plus DevOps & Security * Git, CI/CD pipelines, and automated deployments * Azure DevOps or GitHub Actions * Identity and access management, secrets management, and secure data access * Production monitoring, logging, and incident resolution Preferred Qualifications * Experience migrating legacy data warehouses or on-premises platforms to Azure Databricks. * Experience with Power BI semantic models, datasets, refresh pipelines, and dashboard performance. * Experience with Microsoft Purview or comparable data catalog and governance platforms. * Familiarity with enterprise data architecture, dimensional modeling, and medallion architecture. * Experience working with regulated or security-sensitive enterprise data. * Relevant Azure or Databricks certifications. Success Criteria * Reliable processing and transformation of approximately 50 TB of data. * Effective management and optimization of a data environment containing 10,000+ tables. * Stable, performant data pipelines supporting approximately 300 dashboards. * Improved data quality, pipeline reliability, query performance, and cloud cost efficiency. * Secure and scalable integration of AI/LLM capabilities into enterprise data workflows. ## Description We are seeking an experienced Senior Databricks Engineer with deep expertise in Azure cloud data engineering, large-scale data processing, and enterprise analytics. The ideal candidate will design, develop, optimize, and maintain scalable data platforms handling approximately 50 TB of data, 10,000+ tables, and 300 dashboards., * Design, build, and maintain scalable data engineering solutions using Azure Databricks and Microsoft Azure services. * Develop and optimize ETL/ELT pipelines to process and manage approximately 50 TB of enterprise data. * Manage and optimize data architecture involving 10,000+ tables across multiple source systems. * Support, enhance, and optimize approximately 300 business intelligence dashboards and their underlying data pipelines. * Develop robust data ingestion, transformation, aggregation, and data quality frameworks using PySpark, Spark SQL, and Python. * Implement lakehouse architecture using Delta Lake, Delta tables, and scalable data storage patterns. * Optimize Spark jobs, cluster configurations, partitioning, caching, and query execution to improve performance and control costs. * Integrate Azure Data Factory, Azure Data Lake Storage Gen2, Azure Key Vault, and other relevant Azure services with Databricks. * Architect integration patterns with cloud services (e.g., Azure ADLS Gen2 / AWS S3, Azure Synapse / Snowflake, Kafka, Event Hubs, Power BI / Tableau). * Build reliable data models and curated datasets to support Power BI and other enterprise reporting platforms. * Implement data governance, access controls, lineage, monitoring, and security using Unity Catalog and applicable Azure security services. * Troubleshoot complex production issues, data pipeline failures, performance bottlenecks, and data inconsistencies. * Design and implement AI/LLM-powered data solutions using Claude, Vertex AI, or comparable AI platforms, where applicable. * Develop AI-assisted data discovery, metadata search, natural-language-to-SQL, and analytics automation capabilities. * Collaborate with data architects, BI developers, analytics teams, AI engineers, and business stakeholders across distributed teams. * Establish best practices for CI/CD, version control, automated testing, deployment, and production support.