> Markdown version of [/jobs/ext/1911884-lead-data-ai-engineer](https://www.wearedevelopers.com/jobs/ext/1911884-lead-data-ai-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data & AI Engineer - **Company:** Vanguard - **Location:** Malvern, PA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Amazon Web Services, Data Architecture, Information Engineering, Data Governance, Meta-Data Management, Azure Machine Learning, Software Deployment, Systems Integration, Workflow Management Systems, Data Processing, Cloud Platform System, Multi-Agent Systems, Event Driven Architecture, AWS Glue, Real Time Data, Machine Learning Operations, Virtual Agents, Stream Processing, Data Pipelines, Databricks - **Published:** August 4, 2026 - **Apply:** https://find.jobs/jobs-near-me/apply/ats-redirect/?id=2906665885-2 ## About the Role * 9-12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS). * Hands-on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.). * Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions). * Strong knowledge of lakehouse architecture, Unity Catalog, and data modeling best practices. * Deep experience in data orchestration, monitoring, and scalable AI-driven workflows. ## Description * Architect and build reusable, metadata-driven data and AI engineering frameworks that standardize ingestion, transformation, feature engineering, and AI workflow deployment. Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services to enable scalable, governed, and reusable data products across the organization. * Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing, AI pipelines, and agent data refresh cycles. * Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows. * Integrate Lakeflow for real-time data stream processing, ensuring AI agents are updated and responsive to live data. * Ensure seamless orchestration between AI models and data pipelines, using event-driven architectures for real-time inference and deployment. * Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling, and solutions like Agents on Databricks (Agent-bricks). * Hands-on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations, and agent orchestration frameworks such as AgentOps, AgentBricks, LangGraph, or cloud-native orchestration services. * Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker, Bedrock, or AI orchestration frameworks on AWS. * Ensure robust data governance, metadata management, and AI observability through Unity Catalog, AWS Glue, or custom metadata layers. * Design for scalability and modularity, ensuring AI agents are reusable across multiple business processes. ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) - [How we built an AI-powered code reviewer in 80 hours](https://www.wearedevelopers.com/videos/1511-how-we-built-an-ai-powered-code-reviewer-in-80-hours) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [OLTP in the Lakehouse: Redefining Data for AI Workloads](https://www.wearedevelopers.com/videos/2038-oltp-in-the-lakehouse-redefining-data-for-ai-workloads) - [Azure AI Foundry for Developers: Open Tools, Scalable Agents, Real Impact](https://www.wearedevelopers.com/videos/1541-azure-ai-foundry-for-developers-open-tools-scalable-agents-real-impact) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)