Senior Data Engineer

Experteer Overview As a Senior Data Engineer at Deloitte, you will design and implement end-to-end data solutions that enable client decisions at scale. You’ll collaborate with engagement managers and cross-functional teams to deliver robust ETL/ELT pipelines, governance, and performance optimizations in Azure Databricks environments. You’ll mentor engineers, drive design discussions, and evaluate new technologies to modernize clients’ data platforms. This role offers hands-on impact with minimal travel under the Project Delivery Model. Compensation / Benefits * Design, develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks * Write and tune PySpark / Spark SQL notebooks for large-scale data transformation * Architect end-to-end data solutions across dev UAT prod environments using Unity Catalog * Lead and drive design discussions with client architects and other counterparts * Collaborate with different teams on data contracts and schema agreements * Lead design and optimization of high-volume data pipeline * Define and enforce data engineering standards - naming conventions, partitioning strategies, cluster configurations, Spark tuning * Drive performance optimization - AQE tuning, liquid clustering, broadcast joins, shuffle partition management * Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies * Conduct root cause analysis on production incidents and implement permanent fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree in Computer Science or related IT discipline; or equivalent experience Key requirements *

and and optimization of high-volume data pipeline * Define and enforce data engineering standards - naming conventions, partitioning strategies, cluster configurations, Spark tuning * Drive performance optimization - AQE tuning, liquid clustering, broadcast joins, shuffle partition management * Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies * Conduct root cause analysis on production incidents and implement permanent fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, a implement I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree in Computer Science or related IT discipline; or equivalent experience Key requirements *