> Markdown version of [/videos/1736-minimal-infrastructure-for-real-time-phone-agents-transcripts-in-responses-out?t=573](https://www.wearedevelopers.com/videos/1736-minimal-infrastructure-for-real-time-phone-agents-transcripts-in-responses-out?t=573). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Minimal infrastructure for Real‑Time Phone Agents: transcripts in, responses out

Skip the massive architectural hurdles of voice AI. Deploy real-time, context-aware phone agents in under 100 lines of TypeScript with Twilio Conversation Relay.

- **Speakers:** [Chris Heilmann](https://www.wearedevelopers.com/@chris-heilmann), [Daniel Cranney](https://www.wearedevelopers.com/@daniel-cranney), [Marius Obert](https://www.wearedevelopers.com/@marius-obert), Staff Evangelist At Twilio
- **Event:** WeAreDevelopers LIVE
- **Published:** October 14, 2025
- **Duration:** 29:35
- **URL:** https://www.wearedevelopers.com/videos/1736-minimal-infrastructure-for-real-time-phone-agents-transcripts-in-responses-out

## Summary

Voice interaction is rapidly emerging as a primary interface for generative AI, offering hands-free and highly efficient communication. However, building real-time phone agents traditionally presents massive architectural hurdles. Developers often struggle with assembling complex pipelines connecting automatic speech recognition, large language models, text-to-speech generation, and critically, interruption detection, all while racing to mitigate cascading network latency across disparate API providers.

To eliminate this infrastructure burden, Twilio Conversation Relay abstracts the telecommunications pipeline. By shifting audio transcription and interruption handling to an upstream intermediary layer, developers can focus purely on business and conversational logic. The architecture relies on a persistent WebSocket connection where transcribed caller inputs flow in as text, and the application instantly streams generated text responses back. The relay service handles the audio rendering via integrated speech provider partners like ElevenLabs, dramatically reducing overall round-trip latency.

A complete real-time AI phone agent can realistically be deployed in under 100 lines of modern TypeScript using lightweight runtimes like Deno alongside the OpenAI API. Integrating essential features like conversational memory—achieved by mapping persistent session IDs to specific websocket connections—ensures continuous, context-aware dialogues. Bypassing manual speech modeling and streaming LLM text chunks directly to a telecommunications relay ensures voice applications are highly responsive, handle conversational overlap intuitively, and remain straightforward to maintain.

**Keywords:** real-time voice agents, twilio conversation relay, websocket streaming architecture, llm voice integration, automatic speech recognition, conversational interruption detection, text-to-speech latency, deno websocket server, twiml voice configuration, generative ai phone calls, openai api streaming, websocket session memory, telecommunications infrastructure, elevenlabs voice synthesis

## Chapters

1. **Minimal infrastructure for real-time phone agents** (00:02) — An overview of the minimal components required to build an AI agent accessible via standard phone calls.
1. **Comparing voice and text interaction speeds** (01:56) — Voice inputs offer a hands-free alternative while requiring consideration of cognitive load and synchronous processing limitations.
1. **Challenges of managing voice architecture and latency** (04:48) — Building a voice interface requires careful orchestration of speech recognition, interruption detection, and language model pipelines.
1. **Abstracting complexity with Twilio Conversation Relay** (06:53) — Twilio Conversation Relay manages speech-to-text, text-to-speech, and latency optimizations while decoupling core application logic.
1. **Building a WebSocket server with Deno** (09:33) — A Deno-based WebSocket server parses TwiML instructions, configuring voice models and handling initial connection headers.
1. **Connecting a Twilio phone number to the webhook** (16:41) — Purchasing a local phone number and linking its programmatic routing directly to the deployed web server.
1. **Integrating OpenAI for real-time streaming responses** (19:41) — Streaming large language model outputs minimizes latency by speaking words immediately as language chunks arrive rather than waiting for complete sentences.
1. **Maintaining conversation history and preserving context** (25:10) — Session memory stores historical conversation threads and maps unique request IDs across concurrent connection streams.

## Related Moments

- [Building real-time phone agents with minimal infrastructure](https://www.wearedevelopers.com/videos/1729-wearedevelopers-live-real-time-phone-agents-unsafe-vpns-more) (from "WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More")
- [Addressing latency and architecture in voice agents](https://www.wearedevelopers.com/videos/1729-wearedevelopers-live-real-time-phone-agents-unsafe-vpns-more) (from "WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More")
- [Architecting a web real-time communication stack for agents](https://www.wearedevelopers.com/videos/1641-hello-jarvis-building-voice-interfaces-for-your-llms) (from "Hello JARVIS - Building Voice Interfaces for Your LLMS")
- [Introduction to real-time communication programming interfaces](https://www.wearedevelopers.com/videos/1641-hello-jarvis-building-voice-interfaces-for-your-llms) (from "Hello JARVIS - Building Voice Interfaces for Your LLMS")
- [Building a Deno server for Twilio integrations](https://www.wearedevelopers.com/videos/1729-wearedevelopers-live-real-time-phone-agents-unsafe-vpns-more) (from "WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More")
- [Analyzing voice interface research projects and technical limitations](https://www.wearedevelopers.com/videos/1142-is-the-web-ready-for-voice-user-interfaces) (from "Is the web ready for voice user interfaces?")

## Related Articles

- [Never delegate the understanding](https://www.wearedevelopers.com/magazine/749-never-delegate-the-understanding)
- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)
- [The Web We Broke (And Why AI Agents Are Paying the Price) - AgentCon Berlin](https://www.wearedevelopers.com/magazine/735-the-web-we-broke-and-why-ai-agents-are-paying-the-price-agentcon-berlin)
- [Clone Your Voice with ElevenLabs LLMs and Node](https://www.wearedevelopers.com/magazine/739-clone-your-voice-with-elevenlabs-llms-and-node)

## Related Jobs

- [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio**
- [Twilio's next Senior Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1487390-twilio-s-next-senior-principal-field-architect-ai-agents) at **Twilio**
- [Technical Support Engineer 2](https://www.wearedevelopers.com/jobs/ext/171621-technical-support-engineer-2) at **Twilio**
- [Senior Software Engineer (L3)](https://www.wearedevelopers.com/jobs/ext/1956800-senior-software-engineer-l3) at **Twilio**
- [Artificial Intelligence (AI)](https://www.wearedevelopers.com/jobs/ext/1952055-artificial-intelligence-ai) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**