Senior Data Engineer

Experteer Overview As a Senior Data Engineer, you will design and optimise large-scale data pipelines within Deloitte’s Project Delivery Model. You collaborate with engagement managers and cross-functional teams to deliver end-to-end data solutions across dev, UAT, and prod. You’ll shape data architecture, governance, and performance, while mentoring junior engineers and driving adoption of advanced technologies. This is a hands-on, collaborative role that helps clients modernize data platforms and engineering practices. Compensation / Benefits * Communicate with engagement managers, project team members, and functional/technical teams; escalate where needed * Design, develop and optimise ETL/ELT pipelines using Azure Data Factory and Databricks * Write and tune PySpark / Spark SQL notebooks for large-scale data transformation * Architect end-to-end data solutions across dev, UAT, prod using Unity Catalog * Lead design discussions with client architects and counterparts * Collaborate on data contracts and schema agreements * Lead design and optimisation of high-volume data pipelines * Define and enforce data engineering standards (naming, partitioning, cluster config, Spark tuning) * Drive performance optimisation (AQE tuning, liquid clustering, broadcast joins, shuffle partition management) * Design Databricks cluster policies, autoscaling, and cost optimisation * Conduct root cause analysis on production incidents and implement fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience * Limited immigration sponsorship may be available * Ability to travel 10%, on average, based on the work you do and clients/industries served Key requirements *

on data contracts and schema agreements * Lead design and optimisation of high-volume data pipelines * Define and enforce data engineering standards (naming, partitioning, cluster config, Spark tuning) * Drive performance optimisation (AQE tuning, liquid clustering, broadcast joins, shuffle partition management) * Design Databricks cluster policies, autoscaling, and cost optimisation * Conduct root cause analysis on production incidents and implement fixes * Mentor junior and mid-level engineers through code reviews and pair programming * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O)

• Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience * Limited immigration sponsorship may be available * Ability to travel 10%, on average, based on the work you do and clients/industries served Key requirements *