> Markdown version of [/videos/1650-askui-how-to-leverage-vision-agents-for-test-automation?t=0](https://www.wearedevelopers.com/videos/1650-askui-how-to-leverage-vision-agents-for-test-automation?t=0). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AskUI - How to leverage Vision Agents for Test Automation

Tired of maintaining brittle DOM selectors? Watch AskUI's vision agents bypass code constraints to autonomously execute complex enterprise test suites using only natural language.

- **Speakers:** [Jonas Menesklou](https://www.wearedevelopers.com/@jonas-menesklou)
- **Event:** World Congress 2025
- **Published:** August 20, 2025
- **Duration:** 4:42
- **URL:** https://www.wearedevelopers.com/videos/1650-askui-how-to-leverage-vision-agents-for-test-automation

## Summary

AskUI introduces its vision-driven computer-use agents designed to modernize test automation through natural language instructions. By processing visual inputs directly from a screen, these agents bypass traditional DOM or selector-based constraints to interact seamlessly with interfaces on mobile emulators and native operating systems.

A live demonstration highlights the flexibility of the AskUI framework operating across an Android emulator and a local Mac environment. Utilizing underlying models like Claude, the agent parses natural language commands to complete single-step tasks—such as opening a browser—and complex multi-step workflows, like navigating YouTube or configuring email client applications. The agent dynamically captures screenshots, calculates element positions locally, and executes clicks without manual scripting.

Scaling from ad-hoc operational commands to enterprise test automation, the framework can ingest entire functional test suites formatted as CSV files into a unified prompt. The agent autonomously navigates the full testing sequence across the application lifecycle and generates detailed completion reports, illustrating how organizations can reduce repetitive manual testing with intelligent, self-directing visual automation.

**Keywords:** askui framework, vision agents, computer use agents, test automation, natural language testing, android emulator testing, gui automation workflows, claude vision interaction, prompt-driven execution, csv test suites, automated test reporting, local machine testing, multi-step process automation, visual element targeting

## Chapters

1. **Introduction to AskUI and vision agent capabilities** (00:00) — An introduction to building computer and browser use agents using the AskUI framework.
1. **Demonstrating the Android vision agent with natural language** (00:50) — Controlling an Android emulator via natural language to execute multi-step automation flows.
1. **Running the vision agent on a local macOS environment** (02:29) — Using the agent to automate sequential web searches locally through continuous screenshot analysis.
1. **Automating production test suites using natural language prompts** (03:29) — Running large test cases in a production repository and generating automated execution reports.

## Related Moments

- [Demonstrating automated unit test generation for legacy java](https://www.wearedevelopers.com/videos/100256-can-this-elephant-dance-ibm-bob-and-the-future-of-ai-first-software-development) (from "Can This Elephant Dance? IBM Bob and the Future of AI-First Software Development")
- [Demonstrating a voice-assisted agent for hands-free workflows](https://www.wearedevelopers.com/videos/1538-composable-intelligence-how-henkel-and-microsoft-are-shaping-the-agent-ecosystem) (from "Composable Intelligence: How Henkel and Microsoft Are Shaping the Agent Ecosystem")
- [Automating complex browser workflows with artificial intelligence testing frameworks](https://www.wearedevelopers.com/videos/1288-wearedevelopers-live-scammer-payback-with-python-grok-goes-unhinged-the-future-of-chromium-and-mo) (from "WeAreDevelopers LIVE: Scammer Payback with Python, Grok Goes Unhinged, The Future of Chromium and mo")
- [Handling screen reader simulations using AI agents](https://www.wearedevelopers.com/videos/100170-continuous-accessibility) (from "Continuous Accessibility")
- [Audience questions on visual testing execution and framework differences](https://www.wearedevelopers.com/videos/540-let-s-get-visual-visual-testing-in-your-project) (from "Let's get visual - Visual testing in your project")
- [Introduction to the speaker and visual testing journey](https://www.wearedevelopers.com/videos/540-let-s-get-visual-visual-testing-in-your-project) (from "Let's get visual - Visual testing in your project")

## Related Articles

- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)
- [The Web We Broke (And Why AI Agents Are Paying the Price) - AgentCon Berlin](https://www.wearedevelopers.com/magazine/735-the-web-we-broke-and-why-ai-agents-are-paying-the-price-agentcon-berlin)
- [Liuba Gonta and Yuliya Khadasevic - GitHub Copilot Beyond the Basics - 10 Ways to Elevate Your Coding](https://www.wearedevelopers.com/magazine/490-liuba-gonta-and-yuliya-khadasevic-github-copilot-beyond-the-basics-10-ways-to-elevate-your-coding)

## Related Jobs

- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [AI Operations Manager (all genders)](https://www.wearedevelopers.com/jobs/48263-ai-operations-manager-all-genders) at **envelio**
- [AI Full Stack Engineer](https://www.wearedevelopers.com/jobs/ext/1354435-ai-full-stack-engineer) at **Almedia**
- [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio**
- [Head of AI Applications](https://www.wearedevelopers.com/jobs/ext/1231536-head-of-ai-applications) at **ZEISS Group**