MS Fabric Azure Data Engineer

Sr. Data Engineer MS Fabric, Azure

Core Responsibilities

• Design and build end-to-end data platforms using Microsoft Fabric

• Lakehouse, Warehouse, OneLake, Dataflows Gen2

Develop and optimize Spark workloads using PySpark and SparkSQL

Develop MLOps pipelines for Advanced Analytics & AI

Build scalable ETL/ELT pipelines using:

• Azure Data Factory (ADF)
• MS Fabric Data pipeline
• Dataflow gen 2
• SSIS (on-prem, Azure-SSIS IR, and migration scenarios)

Implement data modeling patterns:

• Medallion (Bronze / Silver / Gold)
• Dimensional modeling (Star/Snowflake)
• Different data file management experience Parquet, JSON, XML

Integrate Microsoft Purview for:

• Data cataloging & classification
• Automated data lineage (ADF, Fabric, SQL, ADLS)

Enforce data security and access controls:

• RBAC, column-level security, masking
• Fabric & Purview policy alignment

Optimize performance, reliability, and cost across Fabric capacities

Implement CI/CD and IaC for data pipelines and governance artifacts

Partner with security, compliance, and BI teams to ensure trusted data delivery

Microsoft Fabric (Must-Have)

• Fabric Data Engineering workloads

• Lakehouse & Warehouse

• OneLake architecture

• Fabric pipelines & notebooks

• Capacity planning and performance optimization

• Advanced PySpark (joins, windows, UDFs, optimization)

• Strong SparkSQL

• Strong MLOps & Feature Engg.

• Partitioning strategies, shuffle tuning, caching

• Large-scale data processing (TB+)

Azure Data Platform

• Azure SQL Database / SQL Server
• Azure Data Factory (ADF)
• SSIS / Azure-SSIS Integration Runtime
• ADLS Gen2

Data Lineage & Security (Microsoft Purview)

• Purview data catalog & scanning
• Automated lineage across ADF, Fabric, SQL, ADLS
• Business glossary management
• Integration with Azure RBAC & security policies