You are not my model anymore - understanding LLM model behavior

Imagine you are a developer and you finished and polished your AI application. Everything seems to work well, then suddenly phone calls or tickets appear with odd behavior you haven’t seen before. User satisfaction goes down, and some things suddenly don’t work anymore.

What happened? The model or other parts of the infrastructure you were using got updated, and you may have missed the memo.

In this session we will look at how even seemingly trivial upgrades that a cloud vendor may make, or a new rollout of a model version can impact your application, and what you need to do to get ahead of the curve.

It sounds like regression testing? Yes, but there are more analytical ways to “understand” what you are dealing with when the underlying AI model changes.