TLDR: This research explores how AI can help people find and use ‘schemas’ – underlying structural patterns – to solve creative problems. It covers past work where schemas were manually identified and applied with AI, and current work on interactive systems like Schemex for discovering schemas and SchemaBuilder for applying them collaboratively with AI, aiming for more transparent and empowering human-AI creative systems.
Creativity often feels like a mysterious process, but beneath every great story, software design, or musical composition lie hidden structural patterns known as schemas. These schemas, like the “hero’s journey” in storytelling or “design patterns” in software, act as guiding frameworks, helping us organize ideas, explore possibilities, and transfer knowledge across different areas. However, discovering these underlying patterns and effectively applying them can be incredibly challenging, especially in new or complex fields.
Imagine trying to find the common narrative thread across movies like Star Wars, The Lord of the Rings, and The Hunger Games. It requires looking beyond the surface details of genre, characters, and settings to identify shared story structures. Similarly, even with a clear schema, turning it into a specific creation, like writing a Harry Potter novel based on the hero’s journey, still demands immense ideation and adaptation.
A New Framework for Human-AI Creative Collaboration
Sitong Wang’s Ph.D. research introduces a groundbreaking framework that leverages human-AI collaboration to make schema discovery and application more accessible for creative problem-solving. The core idea is to help users make sense of examples to abstract these schemas and then turn them into practical, co-creative workflows with AI.
Initially, this research involved manually identifying schemas from expert examples and then designing human-AI workflows to help users apply them. While effective, this process was often slow and labor-intensive, highlighting the need for more intelligent and interactive support. This led to the current focus: building systems that facilitate both schema discovery and application through seamless human-AI teamwork.
From Manual Insights to AI-Powered Assistance
In earlier work, schemas were identified by analyzing successful examples, consulting experts, and using theoretical frameworks. These schemas were then translated into human-AI co-creation workflows. For instance:
- PopBlends: This system helped users create “pop culture blends” – images that creatively combine a product with a pop culture reference. By analyzing examples, a schema was identified focusing on a “connecting concept” between domains. PopBlends then used AI to suggest connecting concepts, scenes, and images, allowing users to blend them. A study showed it helped creators find twice as many creative blends with half the effort.
- ReelFramer: Designed to help journalists turn news articles into short, engaging social media news reels. By studying successful reels, a schema was found involving “narrative framings” (like expository dialogue or comedic analogy) and “premises” (characters, plot, location). ReelFramer guided journalists through exploring these framings and developing scripts, making the process of translating news into reels much easier.
- PodReels: This tool assisted podcasters in creating video teasers for their episodes. The schema identified for effective teasers involved representing the tone, exposing the topic, having a short duration, and high cohesiveness. PodReels used AI to suggest clips based on user criteria, significantly reducing the time and cognitive load for podcasters.
Discovering Schemas with AI: The Schemex Approach
Building on these experiences, the research moved towards facilitating human-AI schema discovery. Discovering schemas is challenging because patterns are often hidden by surface differences, different examples might follow different schemas, and evaluating schema quality is difficult until applied. Schemas also need to balance being specific enough to guide and general enough to transfer.
To tackle this, schema induction is framed as an iterative “sensemaking” process. AI assists in three core steps: clustering examples based on structural similarity, abstracting underlying dimensions and attributes within clusters, and contrastive refinement where generated outputs are compared to real examples to refine the schema. Humans remain central, evaluating coherence and directing the process.
This approach led to Schemex, an interactive visual workflow for schema discovery. Users provide a goal and examples (which can be multimodal, converted to text by AI). Schemex clusters examples, allows users to explore shared features, and then extracts key structural dimensions (e.g., Motivation, Method, Findings for a paper abstract). Users can request inferences about attributes and then apply the derived schema to generate new content. Schemex allows comparison of generated outputs with original examples, highlighting differences and suggesting refinements. This iterative cycle helps users develop coherent and actionable schemas. Users reported richer insights and greater confidence with Schemex compared to baseline tools.
Applying Schemas with AI: The SchemaBuilder Vision
Once a schema is discovered, the next challenge is applying it effectively. Schemas are abstract, but real-world creation requires concrete details. The proposed SchemaBuilder aims to transform schemas into human-AI co-creation workflows. This involves two stages: Workflow Specification, where the system derives a detailed blueprint of steps and decision points, and Workflow Execution, where this specification is translated into a functional prototype using advanced coding agents.
The main hurdle here is creating high-quality workflow specifications. SchemaBuilder plans to extend the “contrastive refinement” idea: it takes a schema and example outputs, generates a candidate workflow, executes it to produce an artifact, and then compares this artifact against “gold” examples to identify gaps. This iterative process of generating, executing, comparing, and refining helps users co-develop reliable, goal-aligned workflows for end-to-end schema-driven creation.
Also Read:
- Designing with AI: How StoryEnsemble Transforms Iteration and Exploration
- Designing AI for Human Well-being: A New Framework for Human-AI Interaction
The Future of Creative Collaboration
The research points to exciting future directions:
- Multimodal Schema Discovery and Application: Extending Schemex to handle visual reasoning, allowing creators to discover and apply visual patterns for designs like posters and covers. This moves beyond static templates to adaptive, structure-aware design workflows.
- Personalized Workflows: Discovering schemas from users’ own creative process histories (e.g., editing sessions) to uncover personal strategies and routines. This could lead to custom tools and agents that reflect an individual’s unique style.
- Co-Agency and Agentic Support: Exploring how AI can provide intelligent assistance for high-effort tasks in schema application, such as context curation and asset manipulation. This involves a dynamic sharing of control between humans and AI, ensuring creative autonomy while leveraging AI’s capabilities.
In conclusion, Sitong Wang’s Ph.D. research offers a compelling vision for advancing human-AI systems. By focusing on making underlying structural patterns – schemas – interactively discoverable and usable, this work opens new possibilities for more insightful, transparent, and empowering collaboration between humans and AI in creative problem-solving. You can read the full research paper here.
