Data Engineer

We are seeking a Data Engineer to support a Microsoft Fabric modernization initiative focused on migrating existing on-premises SQL Stored Procedures workloads into a Fabric Lakehouse environment. This role will be responsible for converting SQL/T-SQL stored procedures into Fabric notebooks, building and orchestrating data pipelines, validating migrated datasets, and creating technical documentation to support ongoing operations. This position will primarily support the Silver Layer within a Medallion Architecture and will play a critical role in ensuring data accuracy, consistency, and scalability throughout the migration effort. Team structure is small, must be highly motivated and self-motivated. Will have access to some SMEs but this role is primarily responsible for the vast majority of the work here. Must be comfortable working in a small team with one other to deliver fast and quick with great quality., Convert existing SQL/T-SQL stored procedures into Microsoft Fabric Notebooks using PySpark and Spark SQL.

Design and build the Silver Layer of a medallion architecture in Fabric using Lakehouse, ensuring parity between Fabric data structures and existing on-premises SQL environments.

Develop and maintain orchestration frameworks using Microsoft Fabric Data Pipelines.

Build automated validation and reconciliation processes to ensure migrated data matches source systems on a 1:1 basis.

Create reusable testing frameworks and validation artifacts to continuously verify data integrity and accuracy.

Support data pipeline monitoring, scheduling, and orchestration activities within Microsoft Fabric.

Partner with architects, engineers, and business stakeholders to ensure successful migration outcomes.

Develop and maintain technical documentation detailing solution design, implementation approaches, validation procedures, and operational support processes.

Analyze and troubleshoot data issues across Fabric notebooks, pipelines, and Lakehouse objects.

5-7+ years of experience as a Data Engineer or similar role.

1-2+ years of hands-on experience with Microsoft Fabric.

Strong expertise in SQL and T-SQL, including:

Stored procedures

Query optimization

Data transformations

SQL-based business logic

Experience migrating or translating SQL-based processing into modern cloud-based data platforms.

Strong hands-on experience with:

PySpark

Spark SQL

Microsoft Fabric Notebooks

Experience developing and managing data pipelines and orchestration workflows.

Deep understanding of:

Lakehouse architecture

Microsoft Fabric data flows

Data Pipelines

Medallion Architecture concepts

Experience building automated data quality, reconciliation, and validation frameworks.

Strong technical documentation and communication skills.

Preferred Qualifications

Experience working with metadata-driven frameworks.

Experience implementing orchestration and scheduling strategies within Microsoft Fabric.

Familiarity with enterprise data migration initiatives.

Experience supporting cloud-based analytics and modern data platform solutions.

Technical Environment

Microsoft Fabric

Fabric Notebooks

Fabric Data Pipelines

Lakehouse Architecture

Medallion Architecture (Silver Layer Focus)

SQL / T-SQL