TLDR: C2-Eval is a new comprehensive benchmark for evaluating the creativity of Foundation Models. It distinguishes between convergent creativity (constrained tasks like code generation) and divergent creativity (open-ended tasks like storytelling), assessing both using Usefulness, Originality, and Surprise (U-O-S) criteria. Key findings include that creativity doesn’t always scale with model size, reasoning capabilities significantly boost creative performance, and explicit creative instructions are particularly effective for enhancing divergent creativity.
The world of artificial intelligence is rapidly evolving, with Foundation Models (FMs) now capable of tasks far beyond what was once imagined. While traditional AI evaluations focused on accuracy, a new frontier is emerging: machine creativity. Long considered a unique human trait, creativity is now seen as a crucial aspect of advanced AI, driving innovation and enabling FMs to act as collaborators in fields like science and art.
However, assessing creativity in these powerful models has been a challenge. Existing methods are often fragmented, using different metrics not consistently rooted in established theories of creativity. To bridge this gap, researchers have introduced C2-Eval, a comprehensive benchmark designed to uniformly evaluate creativity in Foundation Models. This new framework aims to provide a clearer understanding of what makes an AI truly creative.
Understanding C2-Eval: Two Forms of Creativity and Three Core Measures
C2-Eval proposes a unique way to categorize creativity into two complementary forms: convergent and divergent. Convergent creativity involves tasks with constrained solutions, such as generating correct code or answering specific questions. Here, the goal is to find novel and varied solutions within defined boundaries. Divergent creativity, on the other hand, deals with open-ended tasks like storytelling or idea generation, where the model is encouraged to explore a wide range of possibilities.
Across both these types of creativity, C2-Eval uses a fine-grained assessment based on three key components derived from social science theories: Usefulness, Originality, and Surprise (U-O-S). Usefulness refers to how effective and applicable an idea is. Originality measures whether an idea is genuinely new, not just a rephrasing of existing solutions. Surprise captures the unexpectedness or non-obviousness of a solution, distinguishing true ingenuity from straightforward problem-solving.
How Creativity is Measured
For convergent tasks, Usefulness is equated with correctness. Originality is measured by the semantic diversity among correct solutions, while Surprise is linked to the model’s self-reported confidence – lower confidence in a correct answer implies higher surprise. For divergent tasks, where there’s no single “correct” answer, an advanced AI model acts as a judge. This “LLM-as-a-Judge” evaluates responses for coherence and appropriateness (Usefulness), perceived novelty (Originality), and unexpectedness or emotional engagement (Surprise).
Also Read:
- The Evolving Landscape of AI Evaluation: From Simple Recognition to Complex Reasoning
- Unlocking Compositional Generalization in AI Image and Video Creation
Key Insights from Benchmarking Foundation Models
The C2-Eval framework was used to extensively test over 20 leading open-source and proprietary FMs, revealing several fascinating insights:
Top Creative Models: For text-based tasks, Qwen3-235B and o4-mini emerged as the most creative, with Qwen3-235B excelling in long-form content like code generation and storytelling, and o4-mini performing better in shorter outputs like question answering and idea generation. This suggests that open-source models can achieve creativity comparable to, or even surpassing, commercial ones. In vision-based tasks, proprietary models like o4-mini and Claude 3.7 led the pack, demonstrating strong ingenuity in visual question answering and imaginative fluency in visual storytelling, respectively.
Size Isn’t Everything: Surprisingly, creativity doesn’t always increase with model size. The study found instances where smaller models within the same family outperformed larger ones in overall creativity, suggesting an optimal balance rather than a simple scaling up of parameters.
The Power of Reasoning: Models with stronger reasoning capabilities, often those that incorporate a “thinking process,” consistently showed higher creativity scores across all metrics. This indicates that advanced reasoning is a significant driver of creative performance in AI.
Convergent vs. Divergent Strengths: The research highlighted a nuanced relationship between a model’s ability in structured problem-solving (convergent) and open-ended generation (divergent). Models often displayed different strengths in U-O-S dimensions depending on the task type, suggesting that the underlying mechanisms for creativity might be leveraged differently.
The Role of Instructions: Explicit “creative instructions” in prompts generally boosted creativity scores for both convergent and divergent tasks. However, for highly optimized reasoning models in convergent tasks, creative instructions could sometimes lead to less functionally optimal, though more novel, solutions. For divergent tasks, these instructions universally improved creativity, aligning well with their open-ended nature.
In conclusion, C2-Eval offers a robust and systematic way to measure the creativity of Foundation Models, moving beyond traditional accuracy metrics. It provides valuable insights into how different models approach creative tasks, the impact of model architecture and size, and the crucial role of prompting. This benchmark is a significant step towards responsibly guiding the development of truly creative AI. You can read the full research paper for more details here.
