Azure Databricks Data Engineer

• Build batch and streaming pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Implement medallion architecture (Bronze, Silver, Gold), incremental ingestion, and Delta Lake optimization.
• Develop and optimize dimensional data models, SQL queries, and warehouse workloads in Azure Synapse Analytics.
• Orchestrate workflows using Databricks Jobs, Azure Data Factory, and Synapse pipelines.
• Implement Unity Catalog permissions, data lineage, and security controls for sensitive data.
• Optimize Spark workloads, troubleshoot pipeline failures, and monitor production jobs.
• Implement CI/CD, Git version control, and secure access to Azure Data Lake Storage Gen2.
• Support data governance, technical documentation, and integration across enterprise systems.

Seeking an experienced Azure Databricks Data Engineer to build, optimize, and maintain enterprise data pipelines and lakehouse solutions. The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Azure Synapse Analytics, Azure Data Factory, and data governance., Candidates must demonstrate 4 years of experience in the following required areas:

• PySpark, Spark SQL, and Databricks notebooks, including transformations, joins, aggregations, and window functions.
• Delta Lake, ACID transactions, MERGE operations, schema evolution, and table optimization.
• Medallion architecture and incremental ingestion using Auto Loader, Structured Streaming, or Lakeflow/DLT.
• Unity Catalog, permissions, lineage, schemas, and storage access.
• Spark performance tuning, cluster sizing, partitioning, shuffle, skew, and caching.
• Azure Data Factory, Databricks Workflows/Jobs, dependencies, scheduling, and source control.
• Azure Synapse Analytics, dimensional modeling, star schemas, SCD Type 1/Type 2, and SQL performance tuning.
• ADLS Gen2, Parquet/Delta formats, Entra ID, managed identities, and Key Vault.
• Git-based CI/CD using Azure DevOps or GitHub.
• Pipeline monitoring, alerting, troubleshooting, and production support.
• Microsoft lakehouse patterns and interoperability between Azure Synapse and Databricks., Candidates must meet the stated education and experience criteria through a relevant master’s degree, a related bachelor’s degree with additional relevant experience, or an accepted equivalent experience/certification pathway., * Databricks Certified Data Engineer Associate or Professional.
• Microsoft Certified: Azure Data Engineer Associate (DP-203), if already held.
• Power BI connectivity to Synapse, including Import vs. DirectQuery and performance considerations.
• Experience handling sensitive health data and applying HIPAA-aligned security and governance controls.