Lead Data Engineer - Microsoft Fabric

The Lead Data Engineer is responsible for designing, developing, and leading the implementation of scalable data engineering solutions using Microsoft Fabric. The ideal candidate should have strong expertise in modern data engineering practices, ETL/ELT architecture, cloud-based data platforms, data warehousing, and data governance. The role also involves providing technical leadership, ensuring adherence to best practices, and driving the successful delivery of enterprise data solutions., * Lead the design and implementation of enterprise-scale data engineering solutions using Microsoft Fabric.

• Design, develop, and optimize ETL/ELT pipelines for processing.
• Architect and implement metadata-driven data ingestion and transformation frameworks.
• Develop and maintain scalable PySpark notebooks for data processing.
• Design and implement robust data warehousing solutions using Microsoft Fabric.
• Provision and manage Microsoft Fabric workspaces, ensuring appropriate security, governance, and environment configuration.
• Implement and manage data governance, cataloging, metadata management, and data lineage using Microsoft Purview.
• Ensure data quality, security, governance, and compliance, including handling Personally Identifiable Information (PII).
• Collaborate with business stakeholders, solution architects, and reporting teams to understand business requirements and translate them into technical solutions.
• Optimize data pipelines for performance, scalability, reliability, and maintainability.
• Support Power BI reporting by developing optimized data models and curated datasets.
• Work closely with DevOps teams to establish CI/CD pipelines, deployment strategies, and release management processes.

• Bachelor’s degree in computer science, Information Technology, Engineering, or a related discipline.
• 8+ years of experience in Data Engineering with at least 3-4 years of hands-on experience in Microsoft Fabric or similar cloud-based data platforms.
• Proven experience in designing and implementing enterprise ETL/ELT solutions.
• Strong understanding of modern data architecture and cloud-based analytics platforms.
• Hands-on experience with Microsoft Purview for data governance, data cataloging, metadata management, and data lineage.
• Experience provisioning and administering Microsoft Fabric workspaces.
• Good understanding of metadata management concepts and their implementation in relational database systems.
• Familiarity with DevOps methodologies, CI/CD processes, Git-based source control, and deployment automation.
• Strong analytical, problem-solving, leadership, and communication skills.

Technical Skills

• Microsoft Fabric
• Microsoft Purview
• Fabric Workspace Provisioning and Administration
• Fabric Data Pipelines
• PySpark Notebooks
• Metadata-Driven Design
• Metadata Management in Relational Databases
• OneLake
• Fabric Data Warehouse
• Data Engineering and ETL/ELT Architecture
• Data Modeling
• SQL
• Python
• Power BI
• PII Handling and Data Governance
• Azure DevOps
• Git
• CI/CD Pipeline Implementation
• Performance Optimization and Data Pipeline Monitoring

Pay: $120,000.00 - $150,000.00 per year

Experience:

• Microsoft Fabric: 5 years (Required)
• Data Engineer: 8 years (Required)
• ETL: 6 years (Preferred)

$120,000 - $150,000 a year - Full-time, Contract