Fine-Tuning Small Language Models for Agentic AI

Agentic AI promises significant business value, yet many proof-of-concepts fail in production: Large foundation models are expensive, slow, and vulnerable to prompt injection, a risk that escalates when agents interact with untrusted data sources like emails, websites, or documents.

Specialized, fine-tuned Small Language Models (SLMs) with just a few billion parameters can address these challenges. They offer enforceable control through domain constraints and structured outputs, dramatically reduced costs and latency, and inherent security through limited complexity.

This talk explores two approaches to creating specialized agents: Distillation via Synthetic Data (using large models as “teachers” to train smaller “student” models) and Reinforcement Learning from Verifiable Rewards (RLVR).

The core of this presentation focuses on practical evaluation: We present original experiments applying these techniques to real-world agentic applications. If you have a successful agentic AI application, this approach can help make it more efficient without loss of quality, but it doesn’t always work. We evaluate when specialized SLMs can replace or even outperform large models, and when they fall short. Attendees will leave with concrete guidance on where this technique delivers value and where it doesn’t.