TLDR: This research investigates how different tokenization algorithms and preprocessing strategies impact the performance of LLMs and Transformer models in binary code analysis. The study evaluates intrinsic tokenizer properties and extrinsic performance on tasks like masked token prediction and function signature prediction using models like Llama 3.2, BERT, and BART. Key findings indicate that the choice of tokenization algorithm, vocabulary size, and especially preprocessing, significantly influences model outcomes, with optimal strategies being task- and model-dependent. Preprocessing, which normalizes memory addresses and numerical values, consistently improves downstream performance.
Tokenization, the process of breaking down raw data into smaller, manageable units called tokens, is a foundational step in preparing information for Artificial Intelligence (AI) and Machine Learning (ML) models. While widely studied for natural languages and high-level programming languages, its application to assembly code presents unique challenges. Assembly code, with its low-level operations, diverse instruction sets, and non-standardized syntax across different architectures, demands specialized tokenization techniques to accurately capture its structural and semantic complexities.
A recent study delves into this underexplored area, systematically evaluating how different tokenization algorithms and preprocessing methods impact the performance of Large Language Models (LLMs) and Transformer models when applied to binary code analysis. The research, titled “How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis,” was conducted by Ahmed Mostafa, Raisul Arefin, and Samuel Mulder from Auburn University. You can find the full paper here.
The Challenge of Tokenizing Assembly Code
Traditional tokenization methods often fall short for assembly code because they are optimized for natural language or high-level code patterns. Assembly code’s reliance on hardware-specific instructions and its lack of high-level abstractions make it difficult to create structured representations. Furthermore, the prevalence of numerical values, which can vary widely and appear in inconsistent formats, poses a significant hurdle for NLP models primarily trained on textual data.
The Study’s Approach
The researchers conducted a thorough study, analyzing the intrinsic properties of tokenization models (such as efficiency, vocabulary compression, and representational fidelity) and their extrinsic performance on downstream tasks like function signature prediction. They explored preprocessing customization options and pre-tokenization rules specifically tailored for assembly code.
Three state-of-the-art pre-trained models were used for evaluation: the decoder-only LLM Llama 3.2, the encoder-only transformer BERT, and the encoder-decoder model BART. The study utilized three well-established tokenization algorithms from the Hugging Face library: WordPiece, Byte Pair Encoding (BPE), and Unigram. Each algorithm was tested with various vocabulary sizes (3K, 25K, 35K, and 128K for Llama 3.2).
A crucial aspect of their approach was the evaluation of preprocessing. They compared tokenizers trained on two versions of a dataset: default disassembled code and a customized preprocessed version. The preprocessing method involved replacing memory addresses with sequential identifiers (e.g., 0x1FF0 becomes addr1) and converting hexadecimal numeric values to decimal. This normalization aims to reduce variability and eliminate outliers, making the code more structured and interpretable for the models.
Key Findings
The study yielded several important insights:
- Tokenization Efficiency: Intrinsically, Unigram proved to be the most efficient tokenizer in terms of compression, requiring fewer tokens per instruction. WordPiece, conversely, produced the most tokens, offering higher granularity. BPE struck a balance between compression and granularity.
- Impact of Preprocessing: Preprocessing consistently improved tokenization efficiency and slightly enhanced vocabulary alignment across tokenizers. It also generally led to higher accuracy in downstream tasks, especially for smaller to moderate vocabulary sizes.
- Masked Token Prediction (with BERT and Llama 3.2): BERT, designed for masked language modeling, generally outperformed Llama 3.2 in predicting masked tokens. Larger vocabulary sizes and preprocessing improved accuracy for both. Interestingly, BERT’s default tokenizer performed poorly with the preprocessed dataset, highlighting the need for aligning tokenization strategies with preprocessing.
- Function Signature Prediction (with Llama 3.2 and BART): For Llama 3.2, the BPE tokenizer, particularly with larger vocabulary sizes, achieved high accuracy. For BART, a combination of the Unigram tokenizer with a small vocabulary size (3K) and the preprocessed dataset demonstrated the best performance. This suggests that the optimal tokenization strategy is highly dependent on the specific model and task.
Also Read:
- New Research Tackles IoT Vulnerability Detection in Software Issue Reports
- Multilingual Code Learning with AdvFusion: New Insights for Large Language Models
Implications for Binary Code Analysis
The research underscores that the choice of tokenization algorithm, vocabulary size, and dataset representation is critical and must be aligned with the specific AI model and task at hand. Preprocessing machine code to normalize addresses and numerical values consistently enhances performance, making it a valuable step in binary analysis workflows. These findings provide a foundational understanding for effectively representing binary code for NLM-based analysis, ultimately contributing to more robust and scalable solutions in areas like reverse engineering, vulnerability detection, and malware analysis.
Future work aims to expand these investigations to larger datasets, explore alternative tokenization methodologies, scale evaluations to larger language models, and test tokenizers on a wider range of real-world downstream tasks to further validate and generalize these findings.
