TLDR: Researchers have developed LASTIST, a large-scale Korean dataset of 563,299 sentences from political press releases, designed for target-independent stance detection. It addresses the scarcity of non-English and target-agnostic datasets, enabling AI models to identify pro-left or pro-right biases without explicit targets. Experiments show it’s a valuable benchmark, especially for complex target-independent tasks, despite challenges in capturing implicit stances.
Stance detection, a growing field in artificial intelligence, focuses on identifying whether a person or text is in favor of or against a particular topic. However, much of the current research is limited to “target-dependent” stance detection, meaning models are trained to recognize stances towards specific, predefined subjects. This approach often struggles with new or unseen topics. Furthermore, most existing datasets are in English, creating a significant barrier for developing AI models in other languages, especially those with fewer digital resources like Korean.
To address these challenges, researchers DongJae Kim, Yaejin Lee, Minsu Park, and Eunil Park from Sungkyunkwan University have introduced a new dataset called LASTIST (LArge-Scale Target-Independent STance). This innovative dataset aims to bridge the gap in target-independent stance detection and provide a valuable resource for the Korean language.
The LASTIST dataset is unique because it is large-scale, comprising 563,299 labeled Korean sentences. It was meticulously collected from the press releases of major Korean political parties: The Minjoo Party (considered pro-left) and the People Power Party (considered pro-right). By using these distinct sources, the researchers were able to automatically label sentences as either pro-left or pro-right, providing a clear political stance without relying on specific targets within the text.
The data collection process involved several steps. First, articles were scraped from the official websites of both political parties. These documents were then split into individual sentences, as stances are often expressed at the sentence level rather than across an entire document. Boilerplate text, excessively short or long sentences, and other irrelevant content were filtered out. A crucial step involved an active learning framework using a pre-trained Korean BERT model (KPF-BERT) to identify and retain only subjective sentences, as subjectivity is considered a strong indicator of expressed stance.
The final LASTIST dataset contains 394,763 pro-left sentences and 168,536 pro-right sentences. This substantial size makes it suitable for training robust deep learning models.
To demonstrate the dataset’s effectiveness, the researchers conducted experiments using a BERT-based model under two settings:
Target-Independent Setting
In this setting, the full LASTIST dataset was used without providing the model any explicit information about specific targets. The goal was to evaluate how well the model could detect general stances. The results showed that target-independent stance detection is a complex task, indicating the need for more advanced modeling strategies.
Also Read:
- CRADLE BENCH: A New Standard for Detecting Mental Health Crises in AI
- Evaluating Multistep Reasoning in Korean Language Models with Ko-MuSR
Single-Target Setting
For this experiment, a subset of the LASTIST dataset was created, focusing on a single target: party leadership. This allowed for an evaluation of the dataset’s applicability in more narrowly defined scenarios. The model’s performance significantly improved in this setting, highlighting the relative simplicity of single-target detection compared to the target-independent approach. This strong performance also validated the quality and reliability of the labels within the LASTIST dataset.
The LASTIST dataset represents a significant contribution to the field of stance detection. It provides a much-needed large-scale resource for target-independent stance detection, particularly for a low-resource language like Korean. This dataset can facilitate the development of models capable of identifying biases and stances even when explicit targets are not defined, which is crucial for understanding political discourse and public opinion.
While the dataset offers substantial advancements, the authors acknowledge some limitations. The reliance on a subjectivity filter might exclude sentences with more implicit stances, and the active learning framework, while efficient for large datasets, may introduce some internal biases compared to purely manual labeling. Future research could explore human annotations to further enhance label accuracy.
The LASTIST dataset and its associated code are openly available for researchers to use and build upon. You can find more details about this research in the full paper available at this link.
