WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2024 • Aug 22, 2024 • Session details

Performant Architecture for a Fast Gen AI User Experience

Nathaniel Okenwa

Multi-second latency ruins real-time voice apps. Stop relying on naive batch processing. Discover how semantic chunking and edge models combine to deliver instantaneous generative AI experiences.

Pause
Mute Enter Fullscreen
#1 about 6 min

Building a real-time voice translator

A first attempt to create a live conversation translator using basic cloud integrations results in heavy processing delays.

#2 about 4 min

Upgrading the stack with generative APIs

Swapping legacy models for targeted speech generation platforms lowers base latency before deploying deeper architectural improvements.

#3 about 2 min

Lowering pipeline latency with data streaming

Piping inputs between services immediately instead of waiting for full responses eliminates long waterfall bottlenecks.

#4 about 3 min

Balancing speed and context via data chunking

Grouping streamed tokens by sentence boundaries preserves the linguistic context required for high-quality audio generation.

#5 about 6 min

Running edge models to eliminate network trips

Deploying lightweight language models directly alongside the primary processing server eradicates transatlantic API roundtrips.

#6 about 4 min

Using semantic caching and pre-generated audio

Storing pre-computed assets and querying natural language intent via vector databases avoids redundant processing for common interactions.

#7 about 2 min

Minimizing inference time with prompt optimization

Imposing structural constraints on output tokens and parallelizing auxiliary tasks squeezes out the last milliseconds of processing time.

#8 about 1 min

Designing responsive AI loading states

Informing users with progress indicators and transparent application states mitigates the perceived wait time for generative outputs.

#9 about 2 min

Architecture summary and handling output limits

A review of core generative architecture patterns is followed by insights on triggering programmatic buffer flushes.

Matching moments

1:22 min

Leveraging the comprehensive generative artificial intelligence stack

Duan Lightfoot Duan Lightfoot · World Congress 2024

1:10 min

Architecting a web real-time communication stack for agents

Nathaniel Okenwa Nathaniel Okenwa · World Congress 2025

5:28 min

Integrating OpenAI for real-time streaming responses

Chris Heilmann +3 · LIVE

4:03 min

Enhancing conversational output via generative AI models

Xavier Portilla Edo · LIVE

4:16 min

Addressing latency and architecture in voice agents

Chris Heilmann +2 · LIVE

1:38 min

Scaling bottlenecks in generative AI applications

Stan Girard Stan Girard · World Congress 2024

More from World Congress 2024

Beyond the Hype: Real-World AI Strategies Panel thumbnail
24:07
11 months ago

Beyond the Hype: Real-World AI Strategies Panel

Mike Butcher, Jürgen Müller, Katrin Lehmann, Tobias Regenfuss

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation thumbnail
21:17
almost 2 years ago

Carl Lapierre - Exploring Advanced Patterns in Retrieval-Augmented Generation

Carl Lapierre

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI! thumbnail
31:50
about 2 years ago

Coding an Immersive Copilot using Unity / .NET and Azure OpenAI!

Zaid Zaim

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging thumbnail
29:43
about 2 years ago

Building a diverse tech company: The power of Diversity, Equity, Inclusion, and Belonging

Sue Quackenbush

Dapr in Practice thumbnail
24:08
about 2 years ago

Dapr in Practice

marcklefter

Minimising the Carbon Footprint of Workloads thumbnail
27:46
about 2 years ago

Minimising the Carbon Footprint of Workloads

Michael Mueller

Upcoming sessions on this topic

Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 1

Anatomy of an AI Request: Where Latency and Cost Are Really Born

Dan Fu

VP of Kernels at Together AI

Dan Fu
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 7

Designing High-Performance AI APIs: Lessons from Serving Millions of Real-Time Requests

Wayne Liu

Chief Growth Officer and Americas President of Perfect Corp.

Wayne Liu
Open session

World Congress 2026 North America

September 24, 2026 · 14:10–14:40

Stage 5

Edge AI: Running Agentic Intelligence Where Internet Can't Reach

Nitin Eusebius

AWS - Principal Solutions Architect

Nitin Eusebius
Open session

World Congress 2026 North America

September 24, 2026 · 11:00–11:30

Stage 4

From Voice Demo to Enterprise Production

Anahita Havewala, Anuj Gupta

Anahita Havewala
Anuj Gupta
Open session

World Congress 2026 North America

September 23, 2026 · 10:00–17:00

Stage 11

Building Stuff with GenAI - The Open Minded Workshop beyond OpenAI

Andreas Erben

CTO for Applied AI and Metaverse at daenet

Andreas Erben
Open session

World Congress 2026 North America

September 24, 2026 · 16:10–16:40

Stage 1

Stop Blaming the Model: The Art and Science of Context Engineering and Architecture

Lena Hall, Thorsten Hans

Lena Hall
Thorsten Hans

Related videos

Raise your voice! thumbnail
30:59
LIVE • about 6 years ago

Raise your voice!

Lee Boonstra

Chatbots are going to destroy infrastructures and your cloud bills thumbnail
21:21
World Congress 2024 • about 2 years ago

Chatbots are going to destroy infrastructures and your cloud bills

Stan Girard

Hello JARVIS - Building Voice Interfaces for Your LLMS thumbnail
28:13
World Congress 2025 • about 1 year ago

Hello JARVIS - Building Voice Interfaces for Your LLMS

Nathaniel Okenwa

Make it simple, using generative AI to accelerate learning thumbnail
26:54
World Congress 2024 • about 2 years ago

Make it simple, using generative AI to accelerate learning

Duan Lightfoot

WeAreDevelopers LIVE - Local Transcriptions with Open-Source AI thumbnail
1:01:49
LIVE • 3 months ago

WeAreDevelopers LIVE - Local Transcriptions with Open-Source AI

Chris Heilmann, Daniel Cranney, Mattias Geniar

Prompt API & WebNN: The AI Revolution Right in Your Browser thumbnail
30:36
World Congress 2025 • about 1 year ago

Prompt API & WebNN: The AI Revolution Right in Your Browser

Christian Liebel

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.