ULTRON: Your Agent Has No Hands - Real-Time Voice, MCP, and Ten Phones on Stage

I say “unlock my phone and search YouTube for X” out loud. 10 physical Android phone sitting on the table does it - while an orb on screen reacts to my voice in real time. That demo has been watched over 10 million times and garnered over 300 stars on GitHub; this talk is the architecture underneath it, running live on stage.

Three things had to work. First, real-time voice: a WebRTC session to a real-time speech model, where I learned the hard way that the model finishes generating well before the audio finishes playing - so an event-driven state machine gets ahead of the actual voice, and you have to drive state off the output audio itself. Second, hands: a Model Context Protocol (MCP) server bridging the voice agent to ADB and the device’s accessibility tree, with fast deterministic scripted actions for the common cases and an autonomous observe-decide-act-verify loop for everything else. Third, honesty about what I built - including the moment I realised my agent’s system prompt tells it to dismiss Android’s own permission dialogs, and what that means for anyone thinking about shipping this.

Expect real architecture, real code, live devices on stage, and a candid account of where this design is genuinely unsafe and why I drew the line where I did.