TLDR: A new unsupervised method, One-Shot Style Transfer (OSST), uses Large Language Models (LLMs) to analyze writing style for authorship attribution and verification. By measuring how well an LLM can transfer style from one text to another after neutralizing its content, OSST outperforms traditional and existing LLM prompting methods, especially when controlling for topic. The approach scales with model size, shows strong multilingual capabilities, and offers a flexible balance between computational cost and accuracy for identifying and verifying authors.
Understanding who wrote a piece of text is a challenge with wide-ranging applications, from identifying authors of ancient manuscripts to detecting plagiarism and even linking online identities. This field, known as computational stylometry, uses quantitative patterns in writing style to analyze text. Traditionally, methods have relied on hand-crafted features or supervised learning, but these often struggle with distinguishing between an author’s unique style and the topic they are writing about.
Recent advancements in Large Language Models (LLMs) have opened new avenues for authorship analysis. However, existing LLM-based approaches, such as fine-tuning individual models for each author, can be computationally expensive and impractical for tasks like authorship verification where resources might be limited. Other methods that rely on simple prompting of LLMs have shown poor performance, especially with moderately sized models.
Introducing One-Shot Style Transfer (OSST)
A new unsupervised approach, called One-Shot Style Transfer (OSST), leverages the extensive pre-training and in-context learning capabilities of modern LLMs. This method aims to measure how easily the style of one text can be transferred to another, using the LLM’s log-probabilities. The core idea is to first create a ‘neutral’ version of a text by prompting an LLM to rewrite it without any stylistic features. Then, the LLM is tasked with ‘re-styling’ this neutral text back to its original form, using a single example of the target author’s style as a guide. The log-probabilities assigned by the LLM during this re-styling process indicate how well the style of the example text helps in recovering the original text. A higher score suggests a stronger stylistic match.
This innovative method is applied to two main authorship tasks: attribution and verification. Authorship attribution involves determining the most likely author from a set of candidates for a given text. Authorship verification, on the other hand, is a binary decision: determining if a target text was written by a specific author or not. For verification, OSST employs normalization techniques, including a zero-shot baseline (nOSST-ZS) and a random sampling baseline (nOSST-RS), to account for natural variations across different texts.
Also Read:
- Detecting AI-Written Content Through Unique Style Patterns
- Unmasking AI: How Personalized Text Confounds Machine-Generated Content Detectors
Key Findings and Performance
The research demonstrates that OSST-based methods significantly outperform existing LLM prompting approaches of comparable scale. They also achieve higher accuracy than contrastively trained baselines, particularly when efforts are made to control for topical correlations, which often confound other methods. A notable finding is that the performance of OSST consistently improves with the size of the base LLM, suggesting that larger models lead to better results. For authorship verification, the nOSST-RS variant, which uses a fixed set of ‘anchor texts’ for normalization, generally shows improved performance, and even a single anchor example provides a substantial boost over the zero-shot baseline.
Furthermore, OSST methods exhibit robust multilingual performance in authorship attribution tasks, outperforming baselines in non-English settings. This is attributed to the LLMs’ pre-training on large-scale cross-lingual corpora, highlighting OSST’s strong generalization capabilities.
While promising, the study acknowledges limitations, such as the current scale of base models used due to resource constraints, and the challenge of selecting in-distribution anchors for the RS variation in real-world scenarios. Despite these, the OSST approach offers a flexible trade-off between computational cost and accuracy, making it a valuable advancement in unsupervised authorship analysis. You can find the full research paper here: LLMONE-SHOT STYLE TRANSFER FORAUTHORSHIP ATTRIBUTION ANDVERIFICATION.
