TLDR: A research paper explores how sequential fine-tuning improves euphemism detection across five languages (English, Spanish, Chinese, Turkish, Yorùbá) using XLM-R and mBERT models. The study found that training a model first on a high-resource language significantly enhances performance in a second, often low-resource, language. While XLM-R showed greater gains, it was more prone to forgetting the first language; mBERT offered more stable but smaller improvements. The success of transfer was more influenced by pretraining data and dataset characteristics than by linguistic similarity.
Understanding the nuances of language is a complex task for artificial intelligence, especially when it comes to figurative speech like euphemisms. Euphemisms are words or phrases used to soften the impact of something considered harsh, impolite, or taboo, such as saying ‘passed on’ instead of ‘died’. Their meaning can be highly subjective and culturally specific, making them particularly challenging for language models, especially in languages with fewer available resources for training.
A recent study titled “When Does Language Transfer Help? Sequential Fine-Tuning for Cross-Lingual Euphemism Detection” by Julia Sammartino, Libby Barak, Jing Peng, and Anna Feldman from Montclair State University delves into how language models can better detect euphemisms across different languages. The researchers investigated a technique called sequential fine-tuning, comparing it to monolingual (training on one language) and simultaneous (training on two languages at once) approaches.
The study focused on five diverse languages: English, Spanish, Chinese, Turkish, and Yorùbá. They utilized two prominent multilingual transformer models, XLM-RoBERTa (XLM-R) and mBERT, which are known for their ability to capture cross-lingual representations. A key difference between these models is their pretraining coverage; mBERT was pretrained on all five languages, while XLM-R lacked pretraining exposure to Yorùbá.
The core idea behind sequential fine-tuning is to first train a model on a source language (L1) until it reaches peak performance, and then continue training it on a target language (L2) for the same task. The model is then evaluated on both languages to see how knowledge transfers. This approach aims to provide a deeper understanding of how large language models learn abstract figurative language when given the chance to focus on each language individually.
The findings revealed that sequential fine-tuning, particularly when starting with a high-resource language (L1), significantly improved euphemism detection performance in the target language (L2). This was especially true for low-resource languages like Yorùbá and Turkish, which often lack extensive training data. For instance, training on English first and then Yorùbá led to better Yorùbá performance than training Yorùbá alone.
However, the study also highlighted differences between the models. XLM-R achieved larger performance gains in L2 but was more susceptible to ‘catastrophic forgetting,’ where its performance on the initial L1 language significantly dropped after training on L2. This was particularly evident when Yorùbá was the L1 for XLM-R, likely due to its absence from XLM-R’s initial pretraining. In contrast, mBERT showed more stable, though generally smaller, improvements across language pairs and was less prone to catastrophic forgetting, attributed to its more balanced pretraining coverage for all five languages.
Interestingly, the success of language transfer was not primarily determined by how typologically similar the languages were. Instead, factors like the model’s pretraining coverage and the characteristics of the dataset played a more significant role. Strong improvements were observed even between linguistically distant pairs, such as English to Turkish or Yorùbá to Chinese, suggesting that effective transfer can occur beyond close linguistic relationships.
Also Read:
- Bridging the Language Divide: How Code-Switching Improves LLM Performance
- LinguaSafe: Advancing Multilingual Safety Evaluation for Large Language Models
In conclusion, this research demonstrates that sequential fine-tuning is a simple yet effective strategy for enhancing euphemism detection in multilingual models, especially beneficial for languages with limited resources. It underscores the importance of pretraining data and dataset structure in cross-lingual transfer for complex linguistic tasks. For more details, you can refer to the full research paper here.
