TLDR: This research explores how text complexity in pretraining data affects language models. It finds that while simplifying text doesn’t significantly hurt general language understanding, it influences the type of knowledge learned: simpler texts benefit linguistic knowledge tasks, while complex texts are better for world knowledge and entity tracking. Smaller models also perform better on simpler texts.
In the world of artificial intelligence, particularly with language models, it’s widely known that the quality and quantity of data used for training can significantly improve how well these models perform. However, one aspect that hasn’t been explored as much is the role of text complexity. Text complexity refers to how easy or difficult a piece of writing is to read, often judged by things like sentence length, word choice, and sentence structure.
Researchers Dan John Velasco and Matthew Theodore Roque from Samsung R&D Institute Philippines delved into this less-understood area. They aimed to understand how simplifying text, by making sentences shorter, using simpler words, and creating simpler structures—while keeping the core meaning intact—affects language models. Their study, titled “Rethinking the Role of Text Complexity in Language Model Pretraining,” addresses three key questions: how complexity impacts language modeling across different model sizes, whether useful representations can be learned from simple text alone, and how pretraining text complexity influences language understanding tasks.
How the Study Was Conducted
To investigate these questions, the researchers took human-written texts from a high-quality educational dataset called FineWeb-Edu (fwedu_hw). They then used a large language model (Llama 3.1 8B) to create simplified versions of these texts, resulting in a new dataset called fwedu_simp. The simplification process was carefully controlled to ensure that the core content remained the same, but surface-level features like sentence length and word complexity were reduced. For instance, a complex sentence like “As the sunset cast its warm orange glow over Manila Bay, people relaxed on the sideline benches, enjoying the peaceful view of the sunset” might become “The sunset gave Manila Bay a warm, orange light. People sat on the benches and enjoyed the view of the sunset.”
They then pretrained several causal language models, ranging in size from 28 million to 500 million parameters, from scratch. Some models were trained on the original human-written data, and others on the simplified data. After pretraining, these models were evaluated on various language understanding tasks, including finetuning tasks (like BoolQ, MNLI, QQP) and zero-shot tasks (like BLiMP for linguistic knowledge, EWoK for world knowledge, and ARC-Easy for commonsense reasoning).
Key Findings and Insights
The study yielded several interesting results:
- Simplified Data is Indeed Simpler: The researchers confirmed that their simplified corpus (fwedu_simp) was indeed simpler. It had fewer tokens, a smaller vocabulary, lower lexical diversity, and higher Flesch Reading Ease scores (indicating easier readability) compared to the human-written corpus (fwedu_hw). Crucially, a high semantic similarity score showed that the core meaning was preserved.
- Model Size and Complexity Interaction: They found that smaller models (e.g., 28M parameters) showed less degradation in perplexity (a measure of how well a probability model predicts a sample) when trained on simpler texts. This suggests that smaller models might handle lower-complexity text more effectively, and future model design should consider this interaction.
- Minimal Impact on General Language Understanding: For general language understanding tasks, evaluated through finetuning, the text complexity of the pretraining data had little impact. Models trained on both simplified and human-written data performed similarly across various tasks. This suggests that for broad language understanding, the sheer “complexity” of the text might not be the primary driver of performance, but rather the coverage of knowledge within the data.
- Divergent Performance in Zero-shot Tasks: Where text complexity truly made a difference was in zero-shot evaluations, which test a model’s inherent knowledge without further training. Simpler texts (fwedu_simp) seemed to benefit performance on tasks requiring linguistic knowledge (like BLiMP-supplement and PIQA). In contrast, more complex texts (fwedu_hw) favored tasks that required world knowledge and entity tracking (like Entity Tracking, EWoK, and ARC-Easy). This indicates that the type of knowledge learned during pretraining can be influenced by the complexity of the text.
Also Read:
- Tracking the Internal Development of Language Models
- Synthetic Bootstrapped Pretraining: Unlocking Deeper Understanding in Language Models
Conclusion
The research by Velasco and Roque highlights that while simplifying text doesn’t significantly harm performance on general language understanding tasks, it does influence the quality of learned representations in specific ways. Simpler texts can be advantageous for linguistic knowledge, while more complex texts are better for acquiring world knowledge. This study provides valuable insights into the nuanced relationship between text complexity, model capacity, and the types of knowledge language models acquire during pretraining, suggesting that data curation strategies could be tailored based on desired model capabilities.
