TLDR: PACE (Parallel Association Chain Evaluation) is a novel, efficient metric for evaluating the creativity of large language models (LLMs). Inspired by human creativity assessment, it prompts LLMs to generate parallel 20-word association chains from seed words, measuring the semantic distance between words to derive a creativity score. The study found a strong correlation between PACE scores and human-judged creative writing rankings, demonstrating that top LLMs perform comparably to average humans but are still surpassed by creative professionals. Linguistic analysis also revealed that while both humans and LLMs show decreasing concreteness in associations, humans exhibit greater abstractness and diversity in their associative patterns.
Evaluating the creative capabilities of large language models (LLMs) has long been a complex challenge. Traditional methods often grapple with issues like data contamination, where models might have been trained on evaluation data, and the high cost and subjectivity of human assessments. A new research paper introduces an innovative metric called PACE, or Parallel Association Chain Evaluation, designed to overcome these hurdles by drawing inspiration directly from how human creativity is assessed.
The core idea behind PACE is to ask LLMs to generate chains of associated words. This method is rooted in the theory of associative creativity, which suggests that highly creative individuals are adept at forming unconventional connections between disparate concepts. For LLMs, measuring the ‘associative distance’ between words in these chains can reveal their ability to move beyond common semantic links and tap into deeper, more original connections.
The PACE evaluation process is straightforward yet effective. For each ‘seed word’ (e.g., ‘wise’), an LLM is first prompted to generate three distinct associated words. These then become ‘secondary seed words’. Following this, the model is instructed to build a 20-word association chain for each of these secondary seeds, where each new word must associate only with the word immediately preceding it. The semantic distance between words in these chains is then calculated and averaged to produce a creativity score for the model. This parallel generation approach enhances the diversity of associative pathways, offering a broader look into the model’s creative potential.
The researchers selected 110 diverse seed words from the Intercontinental Dictionary Series, ensuring a wide range of semantic domains and word frequencies. They then tested 30 different LLMs, including both open-source and proprietary models, from the Chatbot Arena Leaderboard.
The results were compelling. PACE demonstrated a strong and significant correlation (Spearman’s ρ= 0.739) with the Chatbot Arena Creative Writing rankings, indicating its effectiveness in capturing creative capabilities. Interestingly, this correlation was substantially higher than with general performance rankings, suggesting PACE specifically targets creativity. The metric also proved sensitive enough to differentiate between various versions and sizes of models within the same series, showing that newer generations and larger models generally achieved higher scores.
A fascinating aspect of the study involved comparing LLM performance with human creativity. High-performing LLMs were found to achieve creativity scores comparable to those of average human participants. However, professional humans, such as actors known for their creative abilities, consistently outperformed even the best LLMs. While LLMs showed greater consistency in their minimum performance, highlighting their potential as reliable tools for generating consistent solutions, humans still exhibited a broader range of creative output.
Linguistic analysis further revealed differences in associative patterns. Both humans and LLMs showed a trend of decreasing concreteness in their associations as the chains developed, moving from more tangible to more abstract concepts. However, LLMs consistently produced more concrete associations than humans. Humans also demonstrated a greater diversity in the types of associations, often forming non-semantic links like phonological connections or drawing on personal experiences, which LLMs were less prone to do.
Also Read:
- Calibrating Language Model Open-Endedness with Generation Space Size
- Unpacking LLM Judge Capabilities: Human-Like vs. Super-Consistent AI
In conclusion, PACE offers a simple, scalable, and contamination-free framework for evaluating the creative potential of LLMs. It provides a robust measure of their ability to form deep, unconventional semantic connections, which is a hallmark of genuine creativity. While current leading LLMs can match average human creativity, the unique and diverse associative patterns of highly creative humans still set them apart. This research, detailed in the paper available at https://arxiv.org/pdf/2510.12110, provides a valuable tool for benchmarking and advancing creativity in artificial intelligence.
