Azure Databricks & Agentic AI Architect

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems. The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance., Agentic Data Engineering Leadership Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture. Define autonomous workflows using AI Agents for: Data ingestion Data mapping Schema evolution Data quality remediation Metadata Enrichment Pipeline optimization Root cause analysis Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance. AI-Driven Data Lifecycle (AI-DLC) Lead architecture for AI-enabled Data Development Lifecycle across: Requirement analysis Data modeling Pipeline generation Automated testing Code review Documentation Deployment Monitoring Implement AI copilots to accelerate developer productivity. Enable automated lineage creation and intelligent impact analysis. Lakehouse & Data Platform Architecture Design scalable Lakehouse platforms using: Azure Databricks Delta Lake Unity Catalog ADLS Gen2 Databricks Workflows Delta Live Tables Enterprise GenAI Integration Architect RAG-based solutions using enterprise data assets. Design agent orchestration frameworks using: Azure OpenAI LangGraph Semantic Kernel AutoGen MCP-enabled architectures Build domain-specific AI agents supporting Data Engineering and Analytics teams. AI Governance & Responsible AI Define guardrails for enterprise GenAI adoption. Implement: Prompt governance Observability Cost monitoring Auditability Explainability Security controls Establish governance models for autonomous AI agents. AI-Powered Platform Optimization Design self-healing data pipelines. Implement AI-driven: Incident triage Failure prediction Capacity planning Cost optimization SLA monitoring Enable intelligent workload placement and model routing. Technical Skills Data Platform Azure Databricks Delta Lake Unity Catalog Azure Data Factory AI & Agentic Frameworks Azure OpenAI Knowledge Graph RAG Architecture LangChain LangGraph MCP Protocol Vector Databases AI Agent Orchestration Data Engineering PySpark Spark SQL Python SQL ELT/ETL Modernization DevOps & AI-DLC Azure DevOps GitHub Actions CI/CD MLOps LLMOps Evaluation Frameworks AI Testing Frameworks Leadership Expectations Drive AI-First Data Engineering transformation. Define enterprise patterns, accelerators, and reusable AI agents. Mentor architects, data engineers, and AI engineers. Lead executive conversations on AI adoption, ROI, and transformation roadmaps. Preferred Certifications Databricks Certified Data Engineer Professional Databricks Certified Solution Architect Microsoft Azure Solution Architect (AZ-305) Azure Data Engineer (DP-203) Microsoft Applied Skills Azure OpenAI Generative AI / Agentic AI Certifications Success Metrics 30-50% Data Engineering productivity improvement. Reduction in manual pipeline development effort. Improved data quality and governance compliance. Measurable ROI from Agentic AI adoption. Expansion of reusable AI agents across programs., Job Title: Azure Databricks & Agentic AI Architect Location : Chicago IL Mandatory skills Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering Role Summary: We are s…

• 17 hours ago
• Apply easily, Role: Azure Architect Location: Remote Job type- W2 Contract Exp-10+years Job description Below 10+ years of IT experience with strong Azure architecture expertise Hands-…
• 28 days ago
• Apply easily +

A forward-looking architect with 15+ years of Data & Analytics experience, strong Azure Databricks expertise, and hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems. The candidate should be comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value. Experience: - 16-20 Years Location: - Chicago, IL(Hybrid- 3 days/week in office) Educational Qualifications: -

• Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Technical certification in multiple technologies is desirable. Skills: - Mandatory skills Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering