Nathaniel Okenwa

Hello JARVIS - Building Voice Interfaces for Your LLMS

Can we escape the uncanny valley of conversation? Learn to build voice AIs that handle interruptions and latency like a true conversational partner.

Hello JARVIS - Building Voice Interfaces for Your LLMS
#1about 2 minutes

Introduction to building JARVIS-like voice interfaces

The goal of building a sophisticated voice AI assistant like Iron Man's JARVIS is now more achievable thanks to modern technologies.

#2about 5 minutes

Why natural voice AI has been so difficult

Fictional AI assistants set a high bar for natural voice interaction that early real-world technologies like Siri failed to meet until the arrival of LLMs.

#3about 3 minutes

Navigating the uncanny valley of AI conversations

To avoid the unsettling 'uncanny valley' in voice AI, systems must handle non-linear conversations, interruptions, and the subtle timing of human speech.

#4about 3 minutes

Architecting a composable text-based voice AI stack

A modern voice AI stack combines speech-to-text, an LLM, and text-to-speech, offering more control and better performance than current speech-to-speech models.

#5about 8 minutes

Live demo of handling user interruptions

The demo shows how to implement interruption handling by stopping the AI's audio output and feeding the context of the interruption back into the LLM prompt.

#6about 3 minutes

Using voice interstitials to manage processing delays

Voice interstitials are pre-emptive audio messages that inform the user an action is in progress, preventing the perception of a system failure during long tasks.

#7about 1 minute

Designing AI agents as a constellation of models

An effective voice AI system is not a single monolithic agent but a constellation of smaller, faster models for specific tasks like checking for wake words or playing interstitials.

#8about 2 minutes

Abstracting voice infrastructure with Twilio Conversation Relay

Twilio's Conversation Relay simplifies development by managing the complex audio pipeline, including speech-to-text, text-to-speech, and interruption handling, via a WebSocket API.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Building real-time conversational agents
03:23 MIN

Building real-time conversational agents

Prompt API & WebNN: The AI Revolution Right in Your Browser

Three pillars for integrating LLMs in products
01:47 MIN

Three pillars for integrating LLMs in products

Using LLMs in your Product

Leveraging AI as a new user experience paradigm
04:02 MIN

Leveraging AI as a new user experience paradigm

You are not an AI developer

Adding conversational intelligence with OpenAI and streaming
09:56 MIN

Adding conversational intelligence with OpenAI and streaming

WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More

Building a custom voice AI with WebRTC and Google APIs
00:59 MIN

Building a custom voice AI with WebRTC and Google APIs

Raise your voice!

Building a private, fully on-device smart assistant
02:02 MIN

Building a private, fully on-device smart assistant

Mobile AI Just Got Faster: What’s Coming for Developers on Arm

Why voice is a powerful and efficient AI interface
07:15 MIN

Why voice is a powerful and efficient AI interface

WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More

Using AI for a conversational developer experience
02:20 MIN

Using AI for a conversational developer experience

Platform Engineering untold truths: is just an infrastructure matter?

Featured Partners

Related Videos

See all videos
Three years of putting LLMs into Software - Lessons learned26:30

Three years of putting LLMs into Software - Lessons learned

Simon A.T. Jiménez

Using LLMs in your Product31:12

Using LLMs in your Product

Daniel Töws

AI: Superhero or Supervillain? How and Why with Scott Hanselman25:17

AI: Superhero or Supervillain? How and Why with Scott Hanselman

Scott Hanselman

Bringing the power of AI to your application.32:26

Bringing the power of AI to your application.

Krzysztof Cieślak

Beyond Chatbots: How to build Agentic AI systems24:22

Beyond Chatbots: How to build Agentic AI systems

Philipp Schmid

AI & Ethics28:27

AI & Ethics

PJ Hagerty

On a Secret Mission: Developing AI Agents29:03

On a Secret Mission: Developing AI Agents

Jörg Neumann

Speak, Code, Deploy: Transforming Developer Experience with Voice Commands28:37

Speak, Code, Deploy: Transforming Developer Experience with Voice Commands

Sami Ekblad

Related Articles

View all articles
CH
Chris Heilmann
WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?
Join Scott Hanselman at WWC24 to explore AI's role as a superhero or supervillain. Scott shares his 32 years of experience in software engineering, discusses AI myths, ethical dilemmas, and tech advancements. Engage with his live demos and insights o...
WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?
DC
Daniel Cranney
Stephan Gillich - Bringing AI Everywhere
In the ever-evolving world of technology, AI continues to be the frontier for innovation and transformation. Stephan Gillich, from the AI Center of Excellence at Intel, dove into the subject in a recent session titled "Bringing AI Everywhere," sheddi...
Stephan Gillich - Bringing AI Everywhere

From learning to earning

Jobs that call for the skills explored in this talk.