LLMs introduce non-determinism and a spectrum of quality, making traditional testing methods like unit tests ineffective.
Traditional monitoring dashboards can show a healthy system while silent failures like hallucinations, retrieval errors, and quality drift degrade user experience.
Instead of instrumenting the model itself, capture every interaction around it—like inputs, context, and outputs—into a detailed trace.
Collect critical signals like full prompt context, retrieval scores, model parameters, and user feedback to pinpoint the root cause of quality degradation.
Evals provide a quantitative score for system quality by comparing production outputs against a golden dataset using methods like LLM-as-a-judge or human rating.
Combine observability and evals into a systematic loop of observing, evaluating, debugging, and shipping changes to continuously improve application quality.
Use an open-source library like Langfuse with decorators to automatically create traces and spans for each step in a RAG pipeline, such as retrieval and generation.
Collect explicit signals like ratings and implicit signals like user retries to build an annotated eval dataset that reflects real-world performance.
Traces and evals create a shared source of truth, enabling domain experts, product, engineering, and support teams to collaborate on improving system quality.
Shift focus from whether the system is "up" to whether it is "right" by using traces and evals to continuously measure and improve quality.
Jobs that call for the skills explored in this talk.