While cloud-based AI offers powerful models, a local approach provides better security, lower latency, and no subscription cost by using smaller, specialized models.
Online evaluation uses A/B testing to measure positive signals like code generation and negative signals like user annoyance to validate feature improvements.
Suggestions are validated for semantic correctness by the IDE before being shown to the user, eliminating errors like non-existent variables.
A secondary machine learning model predicts the probability of a suggestion being accepted, filtering out suggestions that are correct but unhelpful.
Using a native C++ inference engine like Llama.cpp enables fast, low-level execution of the language model directly on the user's machine.
Training small, language-specific models in-house is cost-effective and allows for extensive experimentation to optimize performance for local execution.
An offline evaluation pipeline runs the IDE in a headless mode to test hypotheses quickly, pre-selecting the most promising changes for slower A/B tests.
A small, cross-functional team of 10-20 people with diverse skills is an effective structure for delivering complex AI features.
Local AI is a rapidly growing field where success depends on more than just the language model, requiring a focus on security and user experience.
30:24Markus Walker
31:50Cedric Clyburn & Legare Kerrison
31:39Rainer Stropek
32:26Krzysztof Cieślak
30:04Roberto Carratalá & Cedric Clyburn
32:28Hugo Guerrero
28:07Yan Cui
25:17Scott Hanselman
Jobs that call for the skills explored in this talk.
Sonarsource Sa
Geneva, Switzerland
Collaboration Betters The World GmbH
