TLDR: TYPY BENCH is a new benchmark for evaluating how well Large Language Models (LLMs) can infer types in untyped Python codebases. It introduces two new metrics, TYPE SIM (for semantic similarity) and TYPE CHECK (for repository-wide consistency), revealing that while LLMs achieve decent type similarity, they struggle with complex nested types and maintaining consistency across entire projects. The research suggests a need to focus on repository-level consistency in future LLM type inference development.
The world of software development, especially with dynamic languages like Python, often faces a significant hurdle: type inference. This is the process of automatically figuring out what kind of data (like text, numbers, or lists) a variable or expression will hold. While large language models (LLMs) have shown impressive abilities in understanding code, their skill in type inference hasn’t been fully explored.
A new benchmark called TYPY BENCH has been introduced to address this gap. It’s specifically designed to test how well LLMs can infer types across entire Python codebases, not just individual functions or files. This is a crucial distinction because real-world software projects are complex and interconnected.
Innovative Evaluation Metrics
TYPY BENCH introduces two innovative ways to measure performance:
-
TYPE SIM: This metric goes beyond simple exact matches. It looks at the deeper, more nuanced relationships between predicted types and the actual, correct types. For example, it understands that
ListandSequenceare functionally very similar, even if they aren’t identical. This provides a more realistic assessment of how useful a predicted type is. -
TYPE CHECK: This metric evaluates how consistent the inferred types are across an entire codebase. It uses static type checking tools (like Mypy) to see if the predicted types work together without causing errors. This is vital because inconsistent types can lead to significant problems in large software projects, hindering maintenance and refactoring.
The creators of TYPY BENCH evaluated various LLMs on a dataset of 50 high-quality Python repositories. These repositories were chosen for their type coverage, complexity, and variety of domains, including developer tools, machine learning, web/API, and security. The original type annotations were removed from these repositories to create a realistic challenge for the LLMs.
Also Read:
- Code Models Struggle with Imperfect Instructions: A New Study Reveals Robustness Gaps
- Unpacking AI’s Role in Software Security: A Deep Dive into Vulnerability Detection
Key Findings and Challenges
The findings from this extensive evaluation are quite insightful. While LLMs achieved respectable TYPE SIM scores, indicating they can often predict types that are semantically similar to the correct ones, they struggled significantly with complex, nested types. More importantly, they showed considerable issues with type consistency, as revealed by the TYPE CHECK metric. This highlights a key challenge: even if an LLM can guess a type correctly for a single instance, ensuring that type fits seamlessly into the broader codebase remains a major hurdle.
One interesting observation was that while most models struggled with TYPE CHECK, CLAUDE-3.5-SONNET performed best in maintaining type consistency, even though its TYPE SIM score wasn’t the absolute highest. This suggests that some models might be better at producing types that are “fixable” or integrate more smoothly into a larger system.
The research also looked at other factors influencing performance. As type complexity increased (e.g., deeply nested types like List[Dict[str, List[int]]]), LLM performance generally degraded. Predicting less frequent or “rare” types also proved challenging for many models in specific repositories.
Furthermore, the study explored the impact of providing more context to LLMs. When a model like GPT-4O was given the entire repository as context (for smaller repositories), the TYPE CHECK errors decreased significantly. However, this came at the cost of slightly worse TYPE SIM scores, possibly due to the difficulty of extracting and formatting all the correct information from a very long input. This suggests that while more context is beneficial for consistency, handling very long inputs and outputs efficiently is still an area for improvement.
Finally, the researchers noted a potential issue of “data contamination,” where LLMs might have been trained on some of the older repositories in the benchmark. They observed that LLMs performed better on older repositories compared to newer ones, suggesting that using more recent codebases for evaluation can provide a more accurate picture of a model’s true type inference capabilities.
In conclusion, TYPY BENCH provides a valuable framework for evaluating LLMs in Python type inference, shifting the focus from just local accuracy to repository-level consistency. The findings suggest that future research should prioritize improving how LLMs ensure type consistency across entire codebases, rather than solely focusing on individual type prediction accuracy. This work is crucial for developing more effective automated type inference systems that can genuinely assist developers in maintaining high-quality, large-scale Python projects. You can find the full research paper here: TYPY BENCH Research Paper.
