TLDR: A competition organized by JetBrains and Mistral AI explored optimal strategies for collecting context in AI-powered code completion for Python and Kotlin. Participants developed methods to gather relevant code snippets from large repositories, significantly improving the accuracy of fill-in-the-middle completions generated by advanced language models like Codestral, Qwen2.5-Coder, and Mellum. The challenge highlighted the critical role of context quality and the effectiveness of parsing tools combined with information retrieval techniques.
The rapid evolution of AI in software engineering has brought significant advancements, particularly in tools like code completion. However, the effectiveness of these tools heavily relies on their ability to understand and leverage information from an entire project, especially in large and complex codebases. To address this crucial aspect, JetBrains, in collaboration with Mistral AI, organized a challenge focused on optimizing context collection for AI-powered code completion as part of the ASE 2025 conference.
Code completion, a staple in modern integrated development environments (IDEs), predicts and inserts code statements. Contemporary approaches often use a “fill-in-the-middle” formulation, where the AI infills code between a given prefix and suffix. The quality of these completions isn’t solely dependent on the AI model itself but critically on the “context” provided. This context includes file contents, symbol definitions, class hierarchies, and data types gathered from across the entire codebase. A well-chosen context can enable even smaller models to outperform larger, less informed ones.
The challenge aimed to identify the most effective methods for gathering this context. Participants were tasked with developing strategies to collect relevant information from source code repositories to enhance fill-in-the-middle code completions for both Python and Kotlin. A large dataset of real-world code was constructed using permissively licensed open-source projects to ensure realistic scenarios.
How the Competition Worked
The competition was structured into two tracks, one for Python and one for Kotlin, and ran in three phases: Practice, Public, and Private. Participants submitted their collected context for each completion point, rather than the actual neural completion. The competition platform then converted this context into a model-specific prompt for three state-of-the-art neural models: Codestral by Mistral AI, the open-source Qwen2.5-Coder, and JetBrains’ own Mellum. The quality of the completions was evaluated using the chrF metric, a character F-score widely used in machine translation, known for its reliability in assessing code generation quality.
During the public phase, nineteen teams submitted solutions for the Python track and eight for the Kotlin track. Six teams advanced to the private phase, where their solutions were run on a separate, held-out dataset to determine the final winners. The competition specifically focused on the context collection component, assuming all other parts of the code completion pipeline remained constant, as illustrated in the research paper available at arXiv:2510.04349.
Key Strategies from Top Solutions
An analysis of the top solutions revealed several common and effective strategies. Most teams utilized parsing tools to extract complete definitions and symbols mentioned in the code’s prefix and suffix. Classical information retrieval methods, such as BM25, were frequently combined with various heuristics to rank and retrieve the most relevant code chunks. These chunks were then carefully trimmed to fit the context size limitations of the large language models.
For instance, Team NoMoreActimel, ranked first in Python and second in Kotlin, employed query reformulation for retrieval-augmented generation, using embeddings and heuristic boosting to filter context. Team SpareCodeComplete, ranked second in Python and first in Kotlin, leveraged Tree-sitter grammars to extract symbols and query the Zoekt code search engine, ranking results by their distance to the completion point.
Also Read:
- Assessing Code Quality: A Deep Dive into LLM-Generated Code Smells
- FreshBrew: A New Benchmark for Evaluating AI Agents in Java Code Migration
Impact and Future Directions
The challenge successfully explored a diverse solution space for context collection, highlighting practical strategies that significantly improve AI-based code completion. By focusing on this critical, often overlooked, aspect, the competition provided valuable insights and a new dataset for the research community. The findings are expected to contribute to enhancing the coding experience for millions of developers globally, making AI-powered development tools more intelligent and effective in real-world scenarios.
