> Markdown version of [/jobs/ext/48388-data-ai-engineer-microsoft-fabric-experimentation-data-agent-development](https://www.wearedevelopers.com/jobs/ext/48388-data-ai-engineer-microsoft-fabric-experimentation-data-agent-development). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data & AI Engineer - Microsoft Fabric, Experimentation Data & Agent Development

- **Company:** Tata Consultancy Services Limited
- **Location:** Bellevue, WA, United States
- **Salary:** $64,000.0 - $100,000.0
- **Contract:** Permanent contract
- **Skills:** Adobe Analytics, Artificial Intelligence, Data Analysis, Microsoft Azure, Big Data, Data Validation, Data Dictionary, Data Security, Intrusion Detection Systems, Performance Tuning, Role-Based Access Control, Power BI, Azure Machine Learning, Management of Software Versions, Build Management, Adobe, Microsoft Fabric, Pyspark, Core Data, Virtual Agents, Data Pipelines
- **Published:** May 22, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=97f23d53ab59cf9f

## About the Role

Do you have experience in System tuning?

## Description

Core Data Engineering Competencies; Data Concepts & Data Modelling; Digital : Big Data Platforms; data pipelines design; Microsoft fabric data agents; Azure AI Services; AI and ML Integration; Analytical and problem solving skills; Performance tuning and monitoring; Digital : PySpark; ecommerce domain knowledge; Digital : Adobe Analytics; Digital : Customer Analytics; Adobe customer journey analytics; clickstream data
Roles & Responsibilities:

 1. Experimentation data enablement (Silver layer ownership)
Own the design, build, and maintenance of curated Silver-layer datasets in Microsoft Fabric to support experimentation reporting and analysis.
Partner with the Data Reporting/BI team to identify required dimensions, metrics, and joins (visitor/session, variant, campaign/flight, geo, device, channel, funnel steps, conversion events) and ensure these are available in Silver.
Translate experimentation team needs into standardized, reusable data products (tables/views) that can be consumed consistently for scorecards, dashboards, and ad hoc analysis.
Ensure Silver-layer outputs are analysis-ready (cleaned, conformed, deduplicated, and aligned to agreed definitions).

 2. Data gap analysis and assessment
Conduct regular gap assessments between:
experimentation requirements (scorecards/KPIs),
existing Silver layer availability, and
upstream telemetry/source systems.
Identify missing/incorrect fields, inconsistent definitions, data latency issues, or join-key problems; document:
business impact,
severity/priority,
remediation approach,
timelines and dependencies.
Provide recommendations on data model improvements (facts/dimensions, grain, surrogate keys, conformance rules) to reduce recurring data quality issues.

 3. Gold layer requirements and stakeholder requirement gathering
Lead requirement workshops with stakeholders (experimentation, measurement, BI/reporting, engineering) to define Gold layer outputs:
KPI definitions and calculation logic,
experiment attribution rules,
scorecard structure,
segmentation needs and slicing dimensions,
governance and refresh SLAs.
Produce clear functional + technical specifications: source-to-target mappings, data dictionary, metric definitions, validation rules, and acceptance criteria.
Drive alignment on single source of truth definitions to avoid mismatch across CJA/Power BI/scorecards.

 4. Data pipeline engineering (1DS + Fabric pipelines / ADF)
Build and operate robust pipelines using Microsoft Fabric Pipelines and/or ADF to ingest and transform data into Silver and Gold layers.
Understand and work with 1DS (telemetry) pipelines (or equivalent) to ensure required events and attributes flow correctly into Fabric.
Implement reliable orchestration, incremental loads, error handling, and monitoring to meet experimentation reporting timelines.

 5. Data validation and reconciliation (CJA included)
Perform data validation and reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA):
event counts, session/user logic, conversions,
experiment/variant attribution consistency,
time window alignment and filtering rules.
Create validation checks and automated routines for:
missing data detection,
duplicate events,
schema drift,
metric anomalies (sudden drops/spikes),
SRM-supporting signals (where applicable from data).
Document issues and coordinate fixes with upstream owners (telemetry, tagging, product engineering, reporting teams).

 6. Experimentation lifecycle and scorecard readiness
Support the experimentation lifecycle by ensuring datasets are ready for:
pre-launch readiness checks,
launch measurement,
scorecard generation,
ongoing health checks,
post-test learnings/archives.
Enable consistent scorecard outputs by curating:
experiment metadata (test IDs, start/end dates, allocations),
KPI metrics (primary/secondary), and
slicing dimensions required by experimentation stakeholders.

 7. AI agent design & build for experimentation team
Design and build AI-powered agents (Fabric Data Agents / Copilot / Azure OpenAI) to accelerate experimentation workflows, such as:
automated scorecard creation and narrative summaries,
self-serve Q&A over experimentation datasets,
anomaly explanations and investigation guidance,
metric definition assistant / data dictionary lookup,
pipeline health and data quality assistant.
Define the agent's:
scope, personas, and usage scenarios,
grounding data sources (Silver/Gold tables, metadata, documentation),
security model (RBAC, data access boundaries),
evaluation metrics (accuracy, timeliness, adoption).
Partner with experimentation and reporting teams to iterate through pilot feedback rollout.

 8. Documentation, governance, and operational excellence
Maintain documentation for:
dataset definitions (Silver/Gold),
transformation logic,
metric calculation rules,
pipeline design and dependencies,
validation checklists and runbooks.
Establish best practices for:
naming conventions,
semantic consistency,
versioning and backward compatibility,
cost/performance optimization in Fabric.
Provide operational support: monitoring, troubleshooting, incident triage, and continuous improvement.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Data Analytics with Microsoft Fabric: End-to-End Use Case with Data Agents](https://www.wearedevelopers.com/videos/1547-data-analytics-with-microsoft-fabric-end-to-end-use-case-with-data-agents)
- [3x Performance: A Humbling Journey](https://www.wearedevelopers.com/videos/100165-3x-performance-a-humbling-journey)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Dev Digest 164: AI Agents, AI Blindspots and MCP security problems](https://www.wearedevelopers.com/magazine/578-dev-digest-164-ai-agents-ai-blindspots-and-mcp-security-problems)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)
- [ Dev Digest 213: Petrol Prices, Agentic Workflows, AI Skills and CODE100!](https://www.wearedevelopers.com/magazine/718-dev-digest-213-petrol-prices-agentic-workflows-ai-skills-and-code100)
- [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)