Running Modern On-Device AI Solutions on Mobile

On-device AI on mobile is no longer about deploying a single model. In fact, today’s AI and Generative AI solutions rely on deploying and coordinating multiple models, shifting the focus from single model inference to the design and orchestration of complete AI pipelines. This shift is driven by the inherent complexity of modern use cases—such as language models (LMs), text-to-speech (TTS), speech-to-text (STT), or generative audio—which involve multiple models and pre- and post-processing stages, including tokenizers, acoustic models, or vocoders. As a result, these solutions often require diverse data types to achieve the target performance and accuracy, and may span multiple AI inference frameworks. This talk explores how on-device AI has evolved in recent years and how developers can leverage today’s open-source solutions to build sophisticated AI-driven applications. We’ll examine the shift from single-model execution to multi-model, multi-framework systems, unlocking the deployment of new AI capabilities that deliver higher performance and realism, including real-time natural speech synthesis running on a single CPU core. In this talk, we will cover the crucial role of low numerical precision (8-bit, 4-bit, and 2-bit) in enabling on-device AI with runtimes such as LiteRT, ExecuTorch, ONNX Runtime, and llama.cpp The shift from single-model deployment to multi-model pipelines, including strategies for cross-framework orchestration. How modern TTS systems leverage streaming generation to deliver natural and low-latency speech. Through real-world examples—including compact LLMs, advanced TTS pipelines, and streaming audio generation—you’ll learn how to design and deploy GenAI solutions on mobile using today’s open ecosystem. By the end of this session, you will know the options for leveraging advanced AI features on-device, where the real challenge is no longer just running a model but coordinating multiple models efficiently within mobile constraints.