TLDR: This research paper introduces and investigates the problem of multilingual authorship attribution, aiming to identify whether a text was written by a human or a specific Large Language Model across various languages. The study evaluates existing methods on 18 languages and 8 generators, finding that fine-tuned and contrastive learning approaches perform best. It highlights that training on morphologically rich languages like Russian significantly improves cross-lingual transferability, while English is less generalizable. The paper concludes that robust, language-agnostic methods are crucial for accurately attributing authorship in the increasingly multilingual landscape of AI-generated content.
As Large Language Models (LLMs) become increasingly sophisticated, generating text that is almost indistinguishable from human writing, a new challenge emerges: not just telling if a text is machine-generated, but identifying which specific LLM or human created it. This complex task is known as authorship attribution (AA). While earlier efforts focused on simple binary detection (human vs. machine), the rapid proliferation of diverse LLMs necessitates a more granular approach.
Traditionally, authorship attribution research has been confined to a monolingual setting, with English being the most studied language. However, modern LLMs are inherently multilingual, capable of generating content across a vast array of languages. This disparity highlights a critical gap: the need for effective multilingual authorship attribution.
Introducing Multilingual Authorship Attribution
A recent research paper, “Authorship Attribution in Multilingual Machine-Generated Texts,” by Lucio La Cava, Dominik Macko, Robert Moro, Ivan Srba, and Andrea Tagarelli, addresses this crucial gap. The authors formally define and investigate the problem of multilingual authorship attribution, which involves identifying the precise generator (human or one of multiple LLMs) behind a text across diverse languages and writing scripts. They also explore the concept of cross-lingual transferability, assessing how well a model trained on one set of languages can perform on unseen languages.
The Study’s Approach
To conduct their comprehensive study, the researchers utilized the MULTITUDE (v3) dataset, focusing on 18 languages that span multiple language families (e.g., Germanic, Romanic, Slavic, Semitic, Sino-Tibetan) and five writing scripts (Latin, Cyrillic, Arabic, Hanzi, Greek). The dataset included texts from seven different LLMs (Mistral-7B-Instruct-v0.2, OPT-IML-Max-30B, v5-Eagle-7B-HF, Vicuna-13B, Llama-2-70B-Chat-HF, Aya-101, and GPT-3.5-Turbo-0125) alongside human-authored content.
The study evaluated various existing AA methods, categorizing them into statistical approaches (like Fast-DetectGPT, Binoculars, and a combined StatEnsemble) and LLM-based supervised approaches (including fine-tuned encoders like RoBERTa-large and XLM-RoBERTa-large, a contrastive learner called OTBDetector, and a fine-tuned decoder known as mdok). These methods were adapted to handle the complexities of multilingual attribution.
Key Findings and Insights
The research yielded several significant findings:
- Multilingual Suitability: The study found that fine-tuning and contrastive learning approaches, particularly mdok and OTBDetector, demonstrated remarkable adaptability to the multilingual task, consistently achieving high accuracy (F1 scores above 0.9) across most tested languages. Models pretrained on multiple languages (like XLM-RoBERTa-large) generally outperformed monolingual ones. Interestingly, English texts proved challenging to attribute, possibly because LLMs are highly proficient in English, leading to more subtle stylistic differences between generators. Statistical methods, designed for binary detection, struggled significantly in this fine-grained attribution task, especially with non-Latin scripts.
- Cross-lingual Transferability: A crucial insight emerged regarding cross-lingual performance: training models on Russian data had a surprisingly strong positive impact on transferability to other languages, even those from different language families. This is attributed to Russian’s rich morphology and complex syntactic structure, which may compel models to learn deeper, more transferable linguistic patterns compared to languages with simpler structures like English. Conversely, English proved to be the least generalizable training language.
- Impact of LLM Generators: While top-performing models like mdok and OTBDetector could largely learn the unique stylistic fingerprints of different LLMs, some models, such as Mistral, Eagle, and Vicuna, were slightly harder to attribute. Human-written texts, however, remained among the easiest to identify, suggesting that despite LLMs’ fluency, distinct human-specific patterns persist. In cross-lingual scenarios, certain LLMs like OPT-30B and Eagle-7B sometimes became “catch-all” classes, meaning the detectors over-predicted them when uncertain, likely due to their texts having fewer stylistic variations.
Also Read:
- Safeguarding Large Language Models: A Deep Dive into Data Security Risks and Defenses
- Advancing Emotion Understanding with Multimodal AI: A Deep Dive into Language Models
The Path Forward
The paper concludes that while existing methods can be adapted for multilingual authorship attribution, significant challenges remain, particularly in achieving robust cross-lingual transferability across diverse language families and writing scripts. The findings underscore the urgent need for more robust, language-agnostic attribution methods that can effectively handle the linguistic and stylistic diversity present in real-world multilingual scenarios. For more details, you can read the full research paper here.
