Microsoft Fabric Technical lead for data quality and governance

• Design, code, and maintain data quality rules for critical data elements across policy, agent, client, marketing, and customer service domains

• Build reusable validation notebooks in PySpark and SQL against Delta tables in Fabric lakehouses and warehouses

• Profile new sources statistically before they are certified for reporting: distributions, cardinality, null patterns, outliers, drift

• Build source-to-target reconciliation across raw, cleansed, and curated layers

• Trace quality failures through pipeline code and lineage to the originating system or transform

• Validate SCD Type 2 history: overlapping effective dates, gaps, multiple current rows, broken key chains

• Implement and tune rules in Purview Unified Catalog data quality; extend coverage with code where native rules fall short

• Set technical standards, rule templates, and code review criteria for offshore analysts

• Lead definition sessions with business owners and record decisions in the business glossary

• Report governance health and quality trends to the Data Governance Council

Must-Haves (screen on these)

• 5+ years in data quality, data analysis, or analytics engineering with daily hands-on coding

• Expert SQL (T-SQL or Spark SQL): window functions, CTEs, recursive queries, PIVOT/UNPIVOT, query performance on large tables

• Python with PySpark or pandas for profiling and validation, written as reusable code, not one-off scripts

• Has built automated data quality checks that run on a schedule or in a pipeline, with thresholds and alerting

• Source-to-target reconciliation using counts, aggregates, and row-level hash comparisons

• Dimensional modeling depth: grain, surrogate vs. natural keys, SCD Type 2 behavior and its effect on point-in-time queries

• Statistical profiling: distributions, percentiles, standard deviation, and spotting anomalies or drift over time

• Git-based version control for SQL and Python

• Delta Lake or Lakehouse experience (Microsoft Fabric, Databricks, or Synapse)

• Data quality frameworks: Great Expectations, Deequ, dbt tests, Soda, or equivalent

• Regex for pattern validation (policy numbers, IDs, dates)

• Column-level lineage and impact analysis

• REST API work, ideally Purview Data Map or other catalog APIs, for bulk metadata updates

• Reads DAX well enough to confirm a Power BI measure matches its glossary definition

Nice-to-Haves

• Any catalog or governance tool: Purview, Collibra, Alation, Informatica, Unity Catalog

• Azure DevOps pipelines

• Insurance or financial services; annuities a plus

• Experience leading offshore teams