TLDR: A study evaluated six AI text detectors against DeepSeek-generated text, including paraphrased and humanized versions. While some detectors (Copyleaks, QuillBot, GPTZero) performed well, humanized text significantly reduced their accuracy. The study also found that DeepSeek itself, when used with few-shot or Chain-of-Thought prompting, achieved high accuracy in classifying AI vs. human text, highlighting its potential as a self-detector and the need for continuous improvement in external detection tools.
The rapid advancement of large language models, or LLMs, has transformed how we create written content, from academic papers to everyday communications. However, this technological leap has also raised concerns about the authenticity of text, leading to the development of AI detection tools. A recent study delves into the effectiveness of these detectors, particularly when faced with text generated by DeepSeek, a newer LLM, and explores how DeepSeek itself can be prompted to identify AI-generated content.
The Challenge of AI-Generated Text
Previous research on AI text detection primarily focused on well-known LLMs like ChatGPT. This new study addresses a significant gap by evaluating six commonly available AI detection tools against text produced by DeepSeek. The tools examined were AI Text Classifier, Content Detector AI, Copyleaks, QuillBot, GPT-2, and GPTZero. A crucial aspect of the research involved testing these detectors against various forms of DeepSeek-generated text: original, standard paraphrased, and “humanized” paraphrased content. Adversarial attacks, such as paraphrasing, are known to inhibit detectors’ ability to recognize machine-generated text, making this a critical area of investigation.
How the Study Was Conducted
To create a robust dataset, the researchers gathered 49 human-authored question-answer pairs from before the LLM era. DeepSeek-v3 then generated matching responses for these questions. To simulate adversarial conditions, these DeepSeek-generated texts were further processed: some were paraphrased using QuillBot, and others were subjected to DeepThink’s standard and humanized paraphrasing features. This resulted in a comprehensive dataset of 294 samples, including human-written, original DeepSeek, and various forms of altered DeepSeek text.
The detectors were evaluated in several phases. Initially, they were tested on human-written text, then on original DeepSeek text, followed by DeepSeek-paraphrased text, and finally, DeepThink-generated and paraphrased content. The study measured the average accuracy of each detector, noting how well they identified both human and AI-generated content.
Detector Performance: A Mixed Bag
The findings revealed a varied performance among the detectors. When identifying human-written text, QuillBot and Copyleaks achieved near-perfect scores, while AI Text Classifier and GPTZero also performed strongly. However, the landscape shifted when detecting DeepSeek-generated content. GPTZero, Copyleaks, and QuillBot maintained high accuracy, consistently recognizing DeepSeek’s output. In contrast, AI Text Classifier and GPT-2 struggled significantly, showing very low accuracy rates for DeepSeek-generated text.
The study highlighted the vulnerability of detectors to adversarial techniques. While QuillBot, Copyleaks, and GPTZero generally performed well on original and standard paraphrased DeepSeek text, their accuracy notably declined when faced with “humanized” DeepThink-generated text. Humanization proved to be the most effective adversarial attack, significantly reducing the detection accuracy for Copyleaks, QuillBot, and GPTZero.
DeepSeek as its Own Classifier
Beyond evaluating external tools, the research also explored DeepSeek’s inherent capability to classify text as AI-generated or human-written using advanced prompting techniques: few-shot prompting and Chain-of-Thought (CoT) reasoning. Few-shot prompting involves providing the model with a few examples within the prompt to guide its understanding. The study found that with just two or three examples (two-shot and three-shot settings), DeepSeek achieved remarkable 100% accuracy in distinguishing between human and AI text. Even with more examples (four-shot and five-shot), it maintained very high accuracy.
Chain-of-Thought prompting, which guides the model through a step-by-step reasoning process based on linguistic features, also yielded impressive results. DeepSeek, when prompted with CoT, showed high accuracy in identifying both AI-generated and human-written text, demonstrating its potential as a robust internal classifier.
Also Read:
- Detecting AI: A New Framework for Identifying Undetectable Machine-Generated Content
- Spotting Training Data in AI Coders: A New Study Reveals Key Insights
Implications for the Future
This timely research underscores the ongoing challenge of AI content identification as LLM architectures continue to evolve. While some commercial detectors show promise, especially QuillBot and Copyleaks, their effectiveness can be significantly hampered by sophisticated adversarial techniques like humanized paraphrasing. The study also reveals the powerful potential of LLMs like DeepSeek to self-identify AI-generated content through advanced prompting. These findings emphasize the need for continuous improvement in AI detection technologies and careful selection of tools based on specific user requirements in academic, professional, and educational environments. For more detailed insights, you can refer to the full research paper available here.
