WeAreDevelopers Logo WeAreDevelopers
Sign in
For You Scheduled soon LIVE
World Congress 2024 • Aug 22, 2024 • Session details

Performant Architecture for a Fast Gen AI User Experience

Nathaniel Okenwa

Multi-second latency ruins real-time voice apps. Stop relying on naive batch processing. Discover how semantic chunking and edge models combine to deliver instantaneous generative AI experiences.

Pause
Mute Enter Fullscreen
#1 about 6 min

Building a real-time voice translator

A first attempt to create a live conversation translator using basic cloud integrations results in heavy processing delays.

#2 about 4 min

Upgrading the stack with generative APIs

Swapping legacy models for targeted speech generation platforms lowers base latency before deploying deeper architectural improvements.

#3 about 2 min

Lowering pipeline latency with data streaming

Piping inputs between services immediately instead of waiting for full responses eliminates long waterfall bottlenecks.

#4 about 3 min

Balancing speed and context via data chunking

Grouping streamed tokens by sentence boundaries preserves the linguistic context required for high-quality audio generation.

#5 about 6 min

Running edge models to eliminate network trips

Deploying lightweight language models directly alongside the primary processing server eradicates transatlantic API roundtrips.

#6 about 4 min

Using semantic caching and pre-generated audio

Storing pre-computed assets and querying natural language intent via vector databases avoids redundant processing for common interactions.

#7 about 2 min

Minimizing inference time with prompt optimization

Imposing structural constraints on output tokens and parallelizing auxiliary tasks squeezes out the last milliseconds of processing time.

#8 about 1 min

Designing responsive AI loading states

Informing users with progress indicators and transparent application states mitigates the perceived wait time for generative outputs.

#9 about 2 min

Architecture summary and handling output limits

A review of core generative architecture patterns is followed by insights on triggering programmatic buffer flushes.

Matching moments

1:22 min

Leveraging the comprehensive generative artificial intelligence stack

Duan Lightfoot Duan Lightfoot · World Congress 2024

1:10 min

Architecting a web real-time communication stack for agents

Nathaniel Okenwa Nathaniel Okenwa · World Congress 2025

5:28 min

Integrating OpenAI for real-time streaming responses

Chris Heilmann Chris Heilmann +3 · LIVE

4:03 min

Enhancing conversational output via generative AI models

Xavier Portilla Edo · LIVE

4:16 min

Addressing latency and architecture in voice agents

Chris Heilmann Chris Heilmann +2 · LIVE

2:14 min

Optimizing modular voice AI pipelines for speech processing

Anahita Havewala Anahita Havewala +1 · World Congress 2026 North America

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
about 1 year ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
about 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
about 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Livecoding with AI thumbnail
31:39
about 2 years ago

Livecoding with AI

Rainer Stropek

Dapr in Practice thumbnail
24:08
about 2 years ago

Dapr in Practice

marcklefter

Related videos

Raise your voice! thumbnail
30:59
LIVE • over 6 years ago

Raise your voice!

Lee Boonstra

Chatbots are going to destroy infrastructures and your cloud bills thumbnail
21:21
World Congress 2024 • about 2 years ago

Chatbots are going to destroy infrastructures and your cloud bills

Stan Girard

Hello JARVIS - Building Voice Interfaces for Your LLMS thumbnail
28:13
World Congress 2025 • about 1 year ago

Hello JARVIS - Building Voice Interfaces for Your LLMS

Nathaniel Okenwa

From Voice Demo to Enterprise Production thumbnail
31:14
World Congress 2026 North America • 13 days ago

From Voice Demo to Enterprise Production

Anahita Havewala, Anuj Gupta

Make it simple, using generative AI to accelerate learning thumbnail
26:54
World Congress 2024 • about 2 years ago

Make it simple, using generative AI to accelerate learning

Duan Lightfoot

WeAreDevelopers LIVE - Local Transcriptions with Open-Source AI thumbnail
1:01:49
LIVE • 4 months ago

WeAreDevelopers LIVE - Local Transcriptions with Open-Source AI

Chris Heilmann, Daniel Cranney, Mattias Geniar

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.