Senior Data Engineer

Experteer Overview As a Senior Data Engineer, you will design and optimize data pipelines and platforms for enterprise clients within Deloitte’s Project Delivery Model. You will collaborate with cross-functional teams, architect end-to-end data solutions, and mentor junior engineers. You’ll work with cutting-edge technologies to drive scalable, secure data environments and improve production reliability. This role offers hands-on technical impact with a focus on collaboration and client delivery in a consulting context. Compensation / Benefits * Coordinate with Engagement Managers and cross-functional teams, escalating issues as needed * Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks * Develop PySpark / Spark SQL notebooks for large-scale transformations * Architect end-to-end data solutions across dev, UAT, prod with Unity Catalog * Lead design discussions with client architects and counterparts * Collaborate on data contracts and schema agreements * Lead design and optimization of high-volume data pipelines * Define and enforce data engineering standards and configuration best practices * Drive performance optimization, including AQE tuning and partition management * Design Databricks cluster policies and cost optimization strategies * Investigate production incidents and implement fixes * Mentor junior/mid-level engineers through code reviews and pairing * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM templates * Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience * Limited immigration sponsorship may be available * Ability to travel 10% on average Key requirements *

standards * Lead design and optimization of high-volume data pipelines * Define and enforce data engineering standards and configuration best practices * Drive performance optimization, including AQE tuning and partition management * Design Databricks cluster policies and cost optimization strategies * Investigate production incidents and implement fixes * Mentor junior/mid-level engineers through code reviews and pairing * Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs) Tasks * Python, PySpark, Spark SQL, SQL Server * Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor) * Databricks (Delta Lake, Unity Catalog, Workflows) * Apache Airflow * Git / Azure DevOps * Deep Spark internals (DAG optimization, spill analysis, skew handling) * Delta Lake advanced features (time travel, deletion vectors, predictive I/O) * Unity Catalog governance (row/column security, external locations, system tables) * IaC - Terraform, Azure ARM aaa (e.g., * Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience * Limited immigration sponsorship may be available * Ability to travel 10% on average Key requirements *