Beyond Autocomplete: Local AI Code Completion Demystified

The talk is about the details and motivation behind Local AI Code Completion by JetBrains that was released on the 4th of April 2024. I’ll start by talking about the current state of the most popular AI Code Completion tools. We’ll discuss the pros and cons of running language models for code in the cloud versus on the user’s machine. Then we’ll go through the main components of AI Code Completion which is much more than just a language model. We will discuss our approach to offline and online evaluation of AI Code Completion, prompt construction, and smart suggestion filtering. Also, we’ll discuss the rapid progress in efficient language model inference that we’re witnessing nowadays and the perspectives that local AI has given this progress. Finally, I’ll share some insights about the size of the team that made Local AI Code Completion for JetBrains IDEs happen, the model sizes that are currently used in production, and the cost of the models’ training.