TLDR: A new framework, TOKDRIFT, reveals that large language models (LLMs) for code are highly sensitive to minor, semantically irrelevant changes in code formatting or naming conventions. This is because their subword tokenizers, based on statistical frequencies, often misalign with programming language grammar, leading to inconsistent model behavior. The issue originates in early model layers, highlighting the need for grammar-aware tokenization in future code LLMs.
Large language models (LLMs) have become indispensable tools for various programming tasks, from fixing bugs to translating code. However, a recent study introduces a framework called TOKDRIFT, revealing a significant challenge: the way these LLMs “read” code often clashes with how programming languages (PLs) are fundamentally structured. This misalignment, termed “tokenization drift,” can lead to unexpected and inconsistent model behavior, even with minor, semantically inconsequential changes to the code.
The Core Problem: Subwords vs. Grammar
At the heart of this issue is tokenization, the initial step where raw code is broken down into discrete units that an LLM can process. Modern code LLMs typically use subword tokenizers like byte-pair encoding (BPE). These tokenizers learn to merge character sequences based on their statistical frequency in a vast training corpus, which often includes both natural language and programming code. The problem arises because these statistically derived subwords frequently do not align with the clear syntactic boundaries (like keywords, identifiers, and operators) defined by programming language grammar.
For instance, a PL tokenizer would always break down “x+1” into three distinct, stable tokens: an identifier “x”, an operator “+”, and a literal “1”. Adding spaces, like “x + 1”, wouldn’t change these fundamental units. In contrast, a subword tokenizer might split an identifier at an arbitrary point or combine unrelated symbols into a single token, depending on superficial factors like whitespace or identifier naming. This means that two code snippets that are functionally identical to a programmer or compiler might be seen as completely different sequences of tokens by an LLM, potentially altering its understanding and predictions.
Introducing TOKDRIFT: Quantifying the Impact
To measure the real-world impact of this misalignment, researchers developed TOKDRIFT. This framework systematically applies “semantic-preserving rewrite rules” to code. These rules create variants of code snippets that are functionally identical but differ only in their tokenization. Examples include changing the casing style of an identifier (e.g., from camelCase to snake_case) or simply adding a space around an operator (e.g., ). to ) .). The framework then evaluates how nine different code LLMs, including large models with over 30 billion parameters, respond to these subtle changes across three common programming tasks: bug fixing, code summarization, and code translation.
Key Findings: Sensitivity and Its Origins
The results were striking. Even minor tokenization variations caused substantial shifts in model behavior. For example, one of the most performant LLMs tested, Qwen2.5-Coder-32B-Instruct, changed its prediction in over 6% of cases when the input tokenization was altered, and up to 60% under a single rewrite rule. This indicates a significant lack of robustness to changes that should be semantically irrelevant.
A deeper “layer-wise” analysis revealed that this issue originates in the early embedding layers of the LLMs. This suggests that the subword segmentation process itself fails to capture the true grammatical token boundaries right from the start, propagating this initial misunderstanding throughout the model. The study also found that models were particularly sensitive when identifiers were tokenized into different subwords after a rewrite, highlighting the critical role of how variable and function names are processed.
While larger models generally showed slightly less sensitivity than their smaller counterparts, the problem persisted across all evaluated models and scales, indicating that simply increasing model size doesn’t fully resolve the tokenization misalignment.
Understanding the Root Causes
The researchers explored two main root causes. First, they found a correlation between sensitivity and the frequency of words in the LLM’s pre-training corpus. If a rewritten code snippet resulted in a token sequence that was rare in the training data, the LLM was more likely to perform worse. Second, by analyzing the LLM’s internal “hidden states” (which represent its internal comprehension), they observed that for highly sensitive rewrite rules, the model perceived the “before” and “after” versions of the code as semantically different, even though they were functionally identical. This was particularly evident in the middle layers of the model, which are responsible for capturing compressed semantic information.
Also Read:
- Enhancing Math Language Models with Code Structure through Topological Alignment
- AutoCode: Empowering AI to Craft Competitive Programming Challenges
Towards Grammar-Aware LLMs
The findings of the TOKDRIFT study identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation by LLMs. It underscores the urgent need for future code LLMs to adopt “grammar-aware” or “domain-adaptive” tokenization strategies that more faithfully reflect the structure of programming languages. The researchers have open-sourced their framework and data to facilitate further research in this critical area. You can find more details about the TOKDRIFT framework and its findings in the full research paper: TOKDRIFT: When LLM Speaks in Subwords but Code Speaks in Grammar.
