Decision models for developers who use LLMs

Many AI features ask a model to make a decision rather than write text. Is this support ticket urgent? Which tool should the agent call next? Does this review break the content policy? Teams often send these questions to a general-purpose LLM and parse the reply into a label. That works, but it costs more and takes longer than the task needs.

Discriminative models are built for this kind of task. They read text and return one answer from a set you define, such as a label or a score, usually with a probability attached. They cannot answer outside that set. When one fits the task, it costs less per call and replies faster than an LLM. Open models add two more benefits: you can run them on your own infrastructure, and you can retrain them yourself when they get something wrong.

The hard part is knowing when one fits. I use three questions: does the request repeat, is the output structured, and is the set of possible answers finite? I apply them to a few common features, including one where an LLM is the better choice. Then I compare the options available today: fine-tuned BERT-family encoders, newer decision models such as Jev (hosted) and Laya (open weights), and small task-specific models such as the Fern models we build at antelligent.

The last part is a live demo. I run a set of customer support requests through an LLM and two smaller models, and compare accuracy, response time and cost per 10,000 calls. Then I retrain one of the small models on a ticket it got wrong and run that ticket again. The remaining time is for questions.