Data & AI Engineer - Microsoft Fabric, Experimentation Data & Agent Development

Core Data Engineering Competencies; Data Concepts & Data Modelling; Digital : Big Data Platforms; data pipelines design; Microsoft fabric data agents; Azure AI Services; AI and ML Integration; Analytical and problem solving skills; Performance tuning and monitoring; Digital : PySpark; ecommerce domain knowledge; Digital : Adobe Analytics; Digital : Customer Analytics; Adobe customer journey analytics; clickstream data Roles & Responsibilities:

1. Experimentation data enablement (Silver layer ownership) Own the design, build, and maintenance of curated Silver-layer datasets in Microsoft Fabric to support experimentation reporting and analysis. Partner with the Data Reporting/BI team to identify required dimensions, metrics, and joins (visitor/session, variant, campaign/flight, geo, device, channel, funnel steps, conversion events) and ensure these are available in Silver. Translate experimentation team needs into standardized, reusable data products (tables/views) that can be consumed consistently for scorecards, dashboards, and ad hoc analysis. Ensure Silver-layer outputs are analysis-ready (cleaned, conformed, deduplicated, and aligned to agreed definitions).

2. Data gap analysis and assessment Conduct regular gap assessments between: experimentation requirements (scorecards/KPIs), existing Silver layer availability, and upstream telemetry/source systems. Identify missing/incorrect fields, inconsistent definitions, data latency issues, or join-key problems; document: business impact, severity/priority, remediation approach, timelines and dependencies. Provide recommendations on data model improvements (facts/dimensions, grain, surrogate keys, conformance rules) to reduce recurring data quality issues.

3. Gold layer requirements and stakeholder requirement gathering Lead requirement workshops with stakeholders (experimentation, measurement, BI/reporting, engineering) to define Gold layer outputs: KPI definitions and calculation logic, experiment attribution rules, scorecard structure, segmentation needs and slicing dimensions, governance and refresh SLAs. Produce clear functional + technical specifications: source-to-target mappings, data dictionary, metric definitions, validation rules, and acceptance criteria. Drive alignment on single source of truth definitions to avoid mismatch across CJA/Power BI/scorecards.

4. Data pipeline engineering (1DS + Fabric pipelines / ADF) Build and operate robust pipelines using Microsoft Fabric Pipelines and/or ADF to ingest and transform data into Silver and Gold layers. Understand and work with 1DS (telemetry) pipelines (or equivalent) to ensure required events and attributes flow correctly into Fabric. Implement reliable orchestration, incremental loads, error handling, and monitoring to meet experimentation reporting timelines.

5. Data validation and reconciliation (CJA included) Perform data validation and reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA): event counts, session/user logic, conversions, experiment/variant attribution consistency, time window alignment and filtering rules. Create validation checks and automated routines for: missing data detection, duplicate events, schema drift, metric anomalies (sudden drops/spikes), SRM-supporting signals (where applicable from data). Document issues and coordinate fixes with upstream owners (telemetry, tagging, product engineering, reporting teams).

6. Experimentation lifecycle and scorecard readiness Support the experimentation lifecycle by ensuring datasets are ready for: pre-launch readiness checks, launch measurement, scorecard generation, ongoing health checks, post-test learnings/archives. Enable consistent scorecard outputs by curating: experiment metadata (test IDs, start/end dates, allocations), KPI metrics (primary/secondary), and slicing dimensions required by experimentation stakeholders.

7. AI agent design & build for experimentation team Design and build AI-powered agents (Fabric Data Agents / Copilot / Azure OpenAI) to accelerate experimentation workflows, such as: automated scorecard creation and narrative summaries, self-serve Q&A over experimentation datasets, anomaly explanations and investigation guidance, metric definition assistant / data dictionary lookup, pipeline health and data quality assistant. Define the agent’s: scope, personas, and usage scenarios, grounding data sources (Silver/Gold tables, metadata, documentation), security model (RBAC, data access boundaries), evaluation metrics (accuracy, timeliness, adoption). Partner with experimentation and reporting teams to iterate through pilot feedback rollout.

8. Documentation, governance, and operational excellence Maintain documentation for: dataset definitions (Silver/Gold), transformation logic, metric calculation rules, pipeline design and dependencies, validation checklists and runbooks. Establish best practices for: naming conventions, semantic consistency, versioning and backward compatibility, cost/performance optimization in Fabric. Provide operational support: monitoring, troubleshooting, incident triage, and continuous improvement.

Do you have experience in System tuning?

(part of Tata group) 3.93.9 out of 5 stars Bellevue, WA $64,000 - $100,000 a year, Pulled from the full job description

• Pet insurance
• Health insurance
• Vision insurance
• Dental insurance
• Commuter assistance, Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays.