Nathaniel Okenwa

Hello JARVIS - Building Voice Interfaces for Your LLMS

Can we escape the uncanny valley of conversation? Learn to build voice AIs that handle interruptions and latency like a true conversational partner.

Hello JARVIS - Building Voice Interfaces for Your LLMS
#1about 2 minutes

Introduction to building JARVIS-like voice interfaces

The goal of building a sophisticated voice AI assistant like Iron Man's JARVIS is now more achievable thanks to modern technologies.

#2about 5 minutes

Why natural voice AI has been so difficult

Fictional AI assistants set a high bar for natural voice interaction that early real-world technologies like Siri failed to meet until the arrival of LLMs.

#3about 3 minutes

Navigating the uncanny valley of AI conversations

To avoid the unsettling 'uncanny valley' in voice AI, systems must handle non-linear conversations, interruptions, and the subtle timing of human speech.

#4about 3 minutes

Architecting a composable text-based voice AI stack

A modern voice AI stack combines speech-to-text, an LLM, and text-to-speech, offering more control and better performance than current speech-to-speech models.

#5about 8 minutes

Live demo of handling user interruptions

The demo shows how to implement interruption handling by stopping the AI's audio output and feeding the context of the interruption back into the LLM prompt.

#6about 3 minutes

Using voice interstitials to manage processing delays

Voice interstitials are pre-emptive audio messages that inform the user an action is in progress, preventing the perception of a system failure during long tasks.

#7about 1 minute

Designing AI agents as a constellation of models

An effective voice AI system is not a single monolithic agent but a constellation of smaller, faster models for specific tasks like checking for wake words or playing interstitials.

#8about 2 minutes

Abstracting voice infrastructure with Twilio Conversation Relay

Twilio's Conversation Relay simplifies development by managing the complex audio pipeline, including speech-to-text, text-to-speech, and interruption handling, via a WebSocket API.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

Using LLMs in your Product31:12

Using LLMs in your Product

Daniel Töws

AI: Superhero or Supervillain? How and Why with Scott Hanselman25:17

AI: Superhero or Supervillain? How and Why with Scott Hanselman

Scott Hanselman

Bringing the power of AI to your application.32:26

Bringing the power of AI to your application.

Krzysztof Cieślak

Beyond Chatbots: How to build Agentic AI systems24:22

Beyond Chatbots: How to build Agentic AI systems

Philipp Schmid

AI & Ethics28:27

AI & Ethics

PJ Hagerty

On a Secret Mission: Developing AI Agents29:03

On a Secret Mission: Developing AI Agents

Jörg Neumann

Speak, Code, Deploy: Transforming Developer Experience with Voice Commands28:37

Speak, Code, Deploy: Transforming Developer Experience with Voice Commands

Sami Ekblad

From learning to earning

Jobs that call for the skills explored in this talk.