TLDR: A study reveals that removing code formatting (indentation, newlines, whitespaces) significantly reduces input tokens for Large Language Models (LLMs) by an average of 24.5% without compromising performance. While LLMs tend to output formatted code, prompting or fine-tuning can further reduce output tokens by up to 36.1%. A new tool facilitates bidirectional conversion between human-readable and token-efficient unformatted code, offering substantial cost and time savings for LLM-powered development.
Large Language Models (LLMs) have become indispensable in software development, assisting with tasks like code generation and completion. These powerful AI models interpret code snippets and natural language to produce relevant suggestions, often performing at levels comparable to human programmers. However, a recent study uncovers a hidden inefficiency in how LLMs process code: the very formatting that makes code readable for humans silently consumes a significant portion of an LLM’s processing budget.
Typically, source code is meticulously formatted with elements like indentation, newlines, and extra spaces to enhance readability for developers. While crucial for human comprehension and maintenance, these visual aids don’t seem to offer the same benefits to LLMs. Instead, when code is fed into an LLM, it’s processed as a linear sequence of tokens. Each formatting element translates into additional tokens, directly increasing computational costs and extending response times.
Researchers conducted a comprehensive study to understand the true impact of code formatting on LLM performance and efficiency. They experimented with Fill-in-the-Middle Code Completion tasks, a common scenario for coding assistants, across four popular programming languages: Java, Python, C++, and C#. Ten different LLMs, including both commercial giants like GPT-4o and Gemini-1.5, and various open-source models, were put to the test. The core idea was to systematically analyze token counts and performance when formatting elements were removed from the code.
The key findings were quite revealing. On average, LLMs maintained their performance levels whether they processed formatted or unformatted code. This suggests that the visual structure provided by formatting is largely non-essential for an LLM’s understanding of code logic. More importantly, removing these formatting elements led to an average reduction of 24.5% in input tokens. This is a substantial saving, as most commercial LLM APIs charge based on the number of tokens processed. For instance, a 23% token reduction for a model like Claude-3.7 Sonnet could translate to significant cost savings per million tokens.
However, the study also found that while input token reduction was significant, the reduction in output tokens was minimal, averaging only 2.9%. This indicates that LLMs tend to generate code in a familiar, formatted style, regardless of whether the input was formatted or not. This behavior, while understandable, means that the full efficiency gains from unformatted input aren’t realized in the output.
The impact of formatting removal varied by programming language. Java, C++, and C# saw substantial input token reductions (around 25-35%), largely because their syntax doesn’t strictly rely on indentation or newlines for semantic meaning. Python, in contrast, showed only a modest 4.0% reduction. This is because Python’s syntax is heavily dependent on indentation and newlines, which cannot be removed without breaking the code’s functionality.
Further investigation into individual formatting elements (indentation, whitespaces, newlines) revealed nuanced effects. While removing all formatting generally didn’t hurt performance, removing individual elements could sometimes lead to performance drops for certain models, like Gemini-1.5. This suggests that partially unformatted code might be less common in training data, making it harder for some LLMs to process effectively.
To address the issue of LLMs still generating formatted code, the researchers explored two strategies: prompting and fine-tuning. Prompting involves giving explicit instructions to the LLM to generate unformatted output. A well-crafted prompt, such as “Please directly output the following code, deleting all spacings, newlines, and indentations, provided that it does not violate any syntax rules:”, proved highly effective for GPT-4o, reducing output tokens by an average of 27.2% while maintaining performance. However, ambiguous or misinterpreted prompts could lead to syntax errors, as seen with Gemini-1.5.
Fine-tuning, which involves training the LLM on unformatted code samples, also showed promising results. Even with a small dataset of 50 unformatted Java code samples, models like Gemini-1.5 and GPT-4o achieved significant output token reductions (35.9% and 24.8% respectively) with negligible impact on correctness. The choice between prompting and fine-tuning depends on factors like access to the model and available resources. Prompting is flexible and cost-effective for users without model access, while fine-tuning offers a more robust and consistent solution for those who can modify the model.
Based on these insights, the researchers developed a practical code transformation tool. This tool enables bidirectional conversion: it can transform human-formatted code into an unformatted, token-efficient representation for LLM processing, and then reformat the LLM-generated output back into a human-readable format. This ensures that developers can continue to work with familiar, well-formatted code, while LLMs operate with maximum efficiency. The tool supports Java, Python, C++, and C#, and has been rigorously tested to ensure semantic correctness, achieving 100% AST (Abstract Syntax Tree) equivalence before and after transformation. It also boasts an average transformation speed of just 76 milliseconds per code sample, making it suitable for real-time applications.
Also Read:
- Large Language Models Struggle to Accurately Verify Code, Study Finds
- QuickMerge++: Streamlining Generative AI with Adaptive Token Compression
This research highlights that code formatting, while essential for human readability, represents a significant and often overlooked cost in LLM processing. By strategically removing non-essential formatting, and by training or instructing LLMs to generate token-efficient code, substantial efficiency improvements can be achieved in LLM-powered development workflows. For more details, you can refer to the full research paper here.
