Data & AI Engineer Microsoft Fabric, Experimentation Data & Agent Development

We are seeking a highly skilled Data & AI Engineer with strong expertise in Microsoft Fabric, Azure AI Services, data engineering, and AI-powered analytics solutions. The ideal candidate will design and manage scalable experimentation data platforms, build curated Silver/Gold data layers, develop AI-powered data agents, and support enterprise analytics and experimentation workflows.

This role requires hands-on expertise in Microsoft Fabric, PySpark, clickstream analytics, Adobe Customer Journey Analytics (CJA), Azure AI Services, and modern data engineering best practices., * Design, build, and maintain curated Silver-layer datasets in Microsoft Fabric for experimentation analytics and reporting

• Develop reusable data products including standardized tables and views for dashboards, scorecards, and ad hoc analysis
• Ensure datasets are analysis-ready, cleaned, deduplicated, and semantically aligned, * Conduct gap assessments across experimentation KPIs, telemetry systems, and Silver-layer datasets
• Identify data quality, latency, schema, and join-key issues and recommend remediation strategies
• Improve data models using:

• Facts and dimensions
• Conformance rules
• Surrogate keys
• Grain optimization

Gold Layer & Stakeholder Management

• Conduct workshops with BI, reporting, experimentation, and engineering teams
• Define KPI calculations, attribution logic, governance standards, and refresh SLAs
• Create source-to-target mappings, data dictionaries, validation rules, and technical specifications

Pipeline Engineering

• Build and maintain robust data pipelines using:

• Microsoft Fabric Pipelines
• Azure Data Factory (ADF)
• Telemetry/1DS pipelines

• Implement orchestration, incremental loads, monitoring, and error handling mechanisms

Data Validation & Reconciliation

• Validate and reconcile datasets with Adobe Customer Journey Analytics (CJA)
• Build automated validation routines for:

• Missing data
• Duplicate detection
• Schema drift
• Metric anomaly detection

• Coordinate issue resolution with telemetry and engineering teams

AI Agent Development

• Design and build AI-powered data agents using:

• Fabric Data Agents
• Copilot
• Azure OpenAI

• Enable use cases such as:

• Automated scorecards
• Narrative summaries
• Self-service analytics Q&A
• Metric definition assistants
• Data quality monitoring assistants

• Define AI agent architecture, RBAC security, grounding sources, and evaluation metrics

Governance & Operational Excellence

• Maintain documentation for:

• Data transformations
• Pipeline dependencies
• Metric definitions
• Validation rules

• Implement best practices for:

• Naming conventions
• Semantic consistency
• Versioning
• Cost optimization

• Support operational monitoring, troubleshooting, and continuous improvement initiatives

• Strong expertise in:

• Microsoft Fabric
• Data Engineering
• Data Modeling
• Big Data Platforms
• Data Pipeline Design

• Hands-on experience with:

• PySpark
• Azure AI Services
• AI/ML Integration
• Microsoft Fabric Data Agents
• Azure Data Factory (ADF)

• Strong understanding of:

• Clickstream data
• Customer Analytics
• Adobe Analytics
• Adobe Customer Journey Analytics (CJA)
• Experimentation data platforms

• Experience with:

• Data validation and reconciliation
• Performance tuning and monitoring
• Telemetry pipelines
• Data governance and semantic modeling

• Strong analytical and problem-solving skills
• Ecommerce domain experience preferred, * Experience supporting experimentation and customer analytics platforms
• Strong knowledge of ecommerce and digital analytics ecosystems
• Exposure to AI-driven analytics and self-service data platforms
• Strong collaboration and stakeholder communication skills