Senior Data Engineer

Experteer Overview As a Senior Data Engineer, you will design and optimize end-to-end data pipelines and platforms within Deloitte’s AI & Engineering team. You’ll collaborate with engagement managers and cross-functional teams to deliver scalable data solutions on Azure Databricks, Unity Catalog, and Delta Lake. You’ll lead design discussions, drive performance optimization, and mentor junior engineers, shaping data architecture for enterprise clients. This role emphasizes collaboration, technical excellence, and delivering impactful data-driven outcomes. Compensation / Benefits * Design, develop and optimize ETL/ELT pipelines using Azure Data Factory and Databricks * Write and tune PySpark / Spark SQL notebooks for large-scale data transformation * Architect end-to-end data solutions across dev, UAT, prod environments using Unity Catalog * Lead design discussions with client architects and cross-functional teams * Collaborate on data contracts and schema agreements with teams * Lead design and optimization of high-volume data pipelines * Define and enforce data engineering standards (naming, partitioning, cluster configs, Spark tuning) * Drive performance optimization (AQE tuning, liquid clustering, broadcast joins, shuffle partition management) * Design Databricks cluster policies, autoscaling and cost optimization * Conduct root cause analysis on production incidents and implement fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree in Computer Science (or related) * Limited immigration sponsorship may be available * Ability to travel 10% on average Key requirements * broad range of benefits

• design and optimization of high-volume data pipelines * Define and enforce data engineering standards (naming, partitioning, cluster configs, Spark tuning) * Drive performance optimization (AQE tuning, liquid clustering, broadcast joins, shuffle partition management) * Design Databricks cluster policies, autoscaling and cost optimization * Conduct root cause analysis on production incidents and implement fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column aaaaaaa
• external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree in Computer Science (or related) * Limited immigration sponsorship may be available * Ability to travel 10% on average Key requirements * broad range of benefits