Senior Azure Data Engineer

• Develop and maintain scalable ETL/ELT data pipelines
• Extract, transform, clean, and load data from various source systems
• Build data solutions using Azure Databricks, Spark, Delta Lake, Synapse, and ADLS Gen2
• Implement MERGE/Upsert logic using Delta Lake
• Design effective partitioning strategies for large datasets
• Optimize Spark jobs for performance and cost
• Manage data skew, shuffles, caching, partitions, and cluster resources
• Collaborate with development and data teams to deliver reliable data solutions
• Follow best practices for Git, Azure DevOps, and CI/CD where applicable

We are looking for an experienced Senior Azure Data Engineer with 8 10 years of experience in data engineering and strong hands-on expertise in Python, SQL, Apache Spark, PySpark, and Azure Databricks.

The candidate will be responsible for developing and supporting scalable data pipelines, transforming large datasets, and optimizing data processing solutions using Azure cloud technologies.

Must Have Skills

• Strong experience with Python and SQL
• Hands-on experience with Apache Spark, PySpark, and Spark SQL
• Strong experience with Azure Databricks, including:
• Notebooks
• Jobs/Workflows
• Clusters
• Experience with Azure Synapse
• Strong knowledge of Delta Lake
• Experience working with Parquet
• Hands-on experience with ADLS Gen2
• Strong understanding of data engineering and ETL/ELT concepts

Nice to Have Skills

• Experience with Scala
• ETL/ELT development
• Experience with MERGE/Upsert operations
• Knowledge of data partitioning and partition pruning
• Strong Spark performance tuning experience
• Experience handling data skew, shuffles, caching, broadcast joins, and partition optimization
• Experience with Git
• Experience with Azure DevOps and CI/CD