> Markdown version of [/videos/100091-3-ways-to-rebuild-the-data-stack-for-agents](https://www.wearedevelopers.com/videos/100091-3-ways-to-rebuild-the-data-stack-for-agents). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# 3 Ways to Rebuild the Data Stack for Agents

The modern data stack is obsolete for AI agents. Discover how ditching rigid UIs for token-optimized CLIs and programmatic pipelines fundamentally transforms autonomous data querying.

- **Speakers:** [Jordan Tigani](https://www.wearedevelopers.com/@jordan-tigani)
- **Event:** World Congress 2026 Europe
- **Published:** July 9, 2026
- **Duration:** 28:02
- **URL:** https://www.wearedevelopers.com/videos/100091-3-ways-to-rebuild-the-data-stack-for-agents

## Summary

The explosive evolution of LLMs has fundamentally altered the modern data stack, shifting the primary user from humans relying on graphical interfaces to AI agents writing and executing code iteratively. Rather than loosely bolting a "copilot mode" onto existing tools, MotherDuck entirely rebuilt its product surface for autonomous interaction. Because modern LLMs excel at SQL and exploratory data querying, the traditional rigid, low-code UI is now an anti-pattern. Instead, data tooling for agents demands a return to flexible, programmatic access via Python, raw APIs, and highly structured CLIs.

Designing for AI agents required distinct architectural pivots, notably balancing the Model Context Protocol (MCP) against dedicated CLI environments. While MCP excels at seamlessly bypassing OAuth hurdles for non-technical users in web contexts, CLIs drastically reduce token overhead by enabling agents to persist local sandboxes and pass data between steps without routing every byte through the LLM context window. This focus on token optimization is further amplified by adopting the Tune format—a structured hybrid of YAML and CSV—which completely abandons JSON's structural noise, saving computational spend while surprisingly increasing model accuracy.

In this agent-first paradigm, static documentation and dynamic workflows merge directly into the active product interface. Providing Markdown-based "guides" acts as real-time contextual skills, explicitly teaching the LLM essential business logic, available libraries, and gotchas before querying begins. The capabilities of these autonomous pipelines have even prompted legacy BI engines to be phased out in favor of dynamic, agent-generated live dashboards. Ultimately, building for AI agents proves that maintaining backwards compatibility is far more straightforward than with human users; you simply update the documentation, and agents dynamically adapt their behavior on the fly.

**Keywords:** ai agent data pipelines, model context protocol deployment, mcp server integration, command-line interfaces for llms, llm token optimization, tune format vs json, agentic sql generation, duckdb serverless architecture, documentation as product surface, dynamic automated bi frameworks, llm context window management, data stack backward compatibility, motherduck ai architecture, python automation for agents, automated data ingestion workflows

## Chapters

1. **Transitioning from AI co-pilots to AI-native products** (00:03) — The shift from bolting AI onto human interfaces to building for LLMs as primary users.
1. **Overview of MotherDuck and serverless DuckDB infrastructure** (02:24) — MotherDuck runs faster, less expensive cloud queries built on DuckDB for applications.
1. **How LLMs imitate human data discovery behavior** (03:01) — Large language models can now explore datasets and write SQL through trial and error.
1. **Collapsing the modern data stack with AI models** (04:36) — The traditional segmented stack of ingestion and transformation merges into unified vibe-codable tasks.
1. **Replacing complex BI dashboards with live LLM queries** (06:04) — AI models can generate better live embedded visualizations than traditional business intelligence platforms.
1. **Leveraging scale-to-zero hypertenancy for AI agents** (07:48) — Running DuckDB allows every agent instance to independently scale while ensuring seamless cloud scaling.
1. **Why low-code interfaces hinder autonomous AI agents** (08:46) — LLMs perform better when allowed to write Python pipelines instead of using constrained graphical wizards.
1. **Integrating Model Context Protocol for non-technical users** (10:07) — MCP enables instant authentication and cloud application control directly from AI chat systems.
1. **Passing execution context using RAG and markdown guides** (12:28) — Passing exact product constraints and markdown skills reduces redundant parsing steps for inference engines.
1. **Minimizing token usage with the Tune data format** (15:20) — Replacing verbose JSON with Tune formatting drastically decreases parsing tokens and improves system accuracy.
1. **Balancing Model Context Protocol with direct CLI tools** (16:19) — Chaining local command line tools prevents expensive roundtrips and handles raw data better than MCP.
1. **Designing CLI command hierarchies for AI comprehension** (17:35) — Verbose and descriptive command structures allow LLMs to discover capabilities without polluting context windows.
1. **Enabling autonomous agent account signups and provisioning** (19:23) — Structuring deployment commands so agents can provision their own cloud accounts without human intervention.
1. **Constructing multi-step evaluations for agent reliability** (21:08) — Tracking token expenditure and multi-stage intent resolution requires custom built multi-step evaluation harnesses.
1. **End-to-end autonomous data workflow and dashboard demonstration** (22:00) — An LLM independently configures a warehouse, ingests S3 variables, and engineers an interactive dashboard.
1. **Lessons learned in token efficiency and backward compatibility** (26:12) — Redesigning products for agents requires focusing on resource efficiency and embedding changes directly into documentation.
1. **Comparing JSON and Tune formats during audience questions** (27:12) — Structure-agnostic models parse plain Tune format contents more accurately than strictly nested JSON structures.

## Related Moments

- [Leveraging the comprehensive generative artificial intelligence stack](https://www.wearedevelopers.com/videos/969-make-it-simple-using-generative-ai-to-accelerate-learning) (from "Make it simple, using generative AI to accelerate learning")
- [Locating AI agents within the broader machine learning landscape](https://www.wearedevelopers.com/videos/1510-on-a-secret-mission-developing-ai-agents) (from "On a Secret Mission: Developing AI Agents")
- [Inspiration and challenges of scaling AI agent communication](https://www.wearedevelopers.com/videos/1465-event-driven-architecture-breaking-conversational-barriers-with-distributed-ai-agents) (from "Event-Driven Architecture: Breaking Conversational Barriers with Distributed AI Agents")
- [Exploring AI agent usage within the software engineering industry](https://www.wearedevelopers.com/videos/1814-wearedevelopers-live-markdown-liquid-and-checkouts) (from "WeAreDevelopers LIVE - Markdown, Liquid and Checkouts")
- [Building autonomous functions with conversational agent frameworks](https://www.wearedevelopers.com/videos/1624-30-powerful-aws-hacks-in-just-30-minutes-boost-your-developer-productivity) (from "30 powerful AWS hacks in just 30 minutes: Boost your developer productivity")
- [The evolution toward agentic and literate software programming](https://www.wearedevelopers.com/videos/100256-can-this-elephant-dance-ibm-bob-and-the-future-of-ai-first-software-development) (from "Can This Elephant Dance? IBM Bob and the Future of AI-First Software Development")

## Related Articles

- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [From Prototype to Production: Build AI Agents with This Free 4-Course Learning Path](https://www.wearedevelopers.com/magazine/655-from-prototype-to-production-build-ai-agents-with-this-free-4-course-learning-path)

## Related Jobs

- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Product Manager, Agent Platform](https://www.wearedevelopers.com/jobs/ext/277541-principal-product-manager-agent-platform) at **GitHub**
- [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio**