> Markdown version of [/videos/1907-speech-to-speech-ai-models-marius-obert](https://www.wearedevelopers.com/videos/1907-speech-to-speech-ai-models-marius-obert). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Speech-to-Speech AI models - Marius Obert

Marius Obert argues traditional transcription pipelines destroy essential conversational context. Discover how native speech-to-speech models preserve emotional nuances and deploy multimodal voice agents in under 70 lines of code.

- **Speakers:** [Marius Obert](https://www.wearedevelopers.com/@marius-obert)
- **Event:** Coffee With Developers
- **Published:** June 17, 2026
- **Duration:** 9:43
- **URL:** https://www.wearedevelopers.com/videos/1907-speech-to-speech-ai-models-marius-obert

## Summary

Marius Obert discusses the evolving landscape of conversational AI at the Twilio SIGNAL conference, contrasting traditional speech-to-text transcription pipelines with native speech-to-speech models. While pipelined approaches utilizing partners like ElevenLabs or Deepgram afford developers granular control and customizable webhooks, they inherently strip away vital non-verbal context, such as polite hesitations or conversational tone. Conversely, native models like OpenAI's Realtime API and Gemini Live process audio natively, eliminating the transcription bottleneck.<br><br>This native audio processing preserves emotional nuances, enables natural tool calling, and handles seamless multilingual voice continuity without swapping voice personas. Obert notes that while native models function more like a 'magical black box' with limited developer touchpoints, they significantly reduce the overall architectural burden. Developers no longer need to manage separated models for different accents, and the inclusion of multimodal parallel streaming allows contextual text or image inputs to run cheaper alongside raw audio token streams.<br><br>Beyond core architectural tradeoffs, the discussion explores practical telecommunication edge cases, including AI bot fraud prevention and answering machine detection in outbound agentic workflows. Highlighting the Twilio SDK integration with the OpenAI Realtime API, Obert unpacks how complex audio engineering tasks—like managing sample rates and mu-law telephony codecs—are completely abstracted. Developers can now initialize sophisticated, multimodal voice agents natively in under 70 lines of code, saving token costs and reducing conversational latency.

**Keywords:** speech-to-speech ai models, conversational ai pipelines, openai realtime api, gemini live, elevenlabs and deepgram transcription, llm non-verbal context, multimodal token streaming, twilio sdk integration, telephony codec conversion, voice emotion analysis, answering machine detection, ai bot fraud prevention, agentic audio workflows, audio sample rate engineering, voice model language switching

## Chapters

1. **Comparing transcription-based voice pipelines to native models** (00:23) — Routing audio through separate transcription and generation steps strips conversational nuances like hesitations.
1. **Advantages of using native speech-to-speech streaming models** (02:11) — Streaming audio directly through unified models improves latency and maintains consistent voices across languages.
1. **Loss of developer control in black box voice models** (03:21) — Unified voice models abstract away intermediary steps, making it difficult to inject custom logic or webhooks.
1. **Managing token costs and multimodal context in voice agents** (04:24) — Streaming cheaper text tokens alongside audio inputs allows developers to provide rich context efficiently.
1. **Detecting emotional intent directly from voice audio streams** (06:07) — Analyzing raw voice inputs enables models to identify subtle nuances like frustration that text transcriptions lose.
1. **Detecting answering machines and preventing automated voice fraud** (07:11) — Differentiating between human speakers and automated systems remains a complex challenge for continuous outbound calling.
1. **Integrating speech models using the Twilio real-time SDK** (08:30) — New tooling simplifies telephony integration by automatically handling codec translation and sample rate conversions.

## Related Moments

- [Streaming text responses from an AI language model](https://www.wearedevelopers.com/videos/1729-wearedevelopers-live-real-time-phone-agents-unsafe-vpns-more) (from "WeAreDevelopers LIVE – Real-Time Phone Agents, Unsafe VPNs & More")
- [Analyzing voice interface research projects and technical limitations](https://www.wearedevelopers.com/videos/1142-is-the-web-ready-for-voice-user-interfaces) (from "Is the web ready for voice user interfaces?")
- [Enabling live translations with AI voice-to-text pipelines](https://www.wearedevelopers.com/videos/1899-teaching-apis-beyond-the-docs-alex-goldman) (from "Teaching APIs Beyond the Docs - Alex Goldman")
- [Summarizing developer experience and artificial intelligence companions](https://www.wearedevelopers.com/videos/884-forget-developer-platforms-think-developer-productivity) (from "Forget Developer Platforms, Think Developer Productivity!")
- [Challenges of managing voice architecture and latency](https://www.wearedevelopers.com/videos/1736-minimal-infrastructure-for-real-time-phone-agents-transcripts-in-responses-out) (from "Minimal infrastructure for Real‑Time Phone Agents: transcripts in, responses out")
- [Enhancing conversational intent through modern large language models](https://www.wearedevelopers.com/videos/1641-hello-jarvis-building-voice-interfaces-for-your-llms) (from "Hello JARVIS - Building Voice Interfaces for Your LLMS")

## Related Articles

- [WWC24 Talk - Scott Hanselman - AI: Superhero or Supervillain?](https://www.wearedevelopers.com/magazine/469-wwc24-talk-scott-hanselman-ai-superhero-or-supervillain)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)

## Related Jobs

- [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio**
- [Twilio's next Senior Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1487390-twilio-s-next-senior-principal-field-architect-ai-agents) at **Twilio**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO**
- [Artificial Intelligence (AI)](https://www.wearedevelopers.com/jobs/ext/1952055-artificial-intelligence-ai) at **Twilio**