TLDR: This research introduces a new approach to personalized art therapy by using music preferences to recommend visual art. Traditional art therapy recommenders rely only on visual input, but this study explores how music, known for its emotional impact, can better capture user preferences. The researchers developed three new cross-domain recommendation methods—Mozart, Haydn, and Salieri—that bridge music and visual art. A large study with 200 participants showed that these music-driven methods are as effective as, and sometimes even outperform, traditional visual-only approaches in improving mood and providing therapeutically relevant art recommendations.
Art therapy (AT) is a well-established method that helps individuals process emotions and recover through creative expression. Recently, Visual Art Recommender Systems (VA RecSys) have emerged to personalize therapeutic artwork recommendations, showing great promise. However, these systems typically rely on visual information alone to understand user preferences, which can limit their ability to fully capture the range of emotional responses during the preference selection process.
Previous research has indicated that music can evoke unique emotional reflections. This insight opens up an exciting opportunity for Cross-Domain Recommendation (CDR) to significantly enhance personalization in art therapy. Since CDR had not been explored in this specific context, a new study proposes a family of CDR methods for AT that are based on eliciting preferences through music.
The researchers developed three novel affect-aware CDR algorithms: Mozart, Haydn, and Salieri. Mozart focuses on affect-aware contrastive alignment, mapping the emotional properties of music and paintings into a shared space for semantic retrieval of therapeutic content. Haydn uses an affective space search, leveraging the valence and arousal labels of each modality to create a similarity matrix for recommendations. Salieri employs multimodal alignment, utilizing large language models (LLMs) and vision-language models (VLMs) to extract semantic representations from both music and visual art, facilitating cross-modal mapping.
To test the effectiveness of these music-driven approaches, a large-scale study was conducted with 200 users. The study demonstrated that music-driven preference elicitation was effective, even outperforming the classic visual-only elicitation approach in some aspects. The source code, data, and models from this research are openly available.
The study utilized two main datasets: the DEAM (Database for Emotional Analysis of Music) dataset for instrumental music excerpts, annotated with valence and arousal values, and the WikiArt Emotions Dataset for paintings, which is annotated with multiple emotions and their intensities. A crucial preprocessing step involved converting the different emotional labels from both datasets into a uniform valence-arousal (V-A) scale, ensuring consistency for cross-modal comparison.
Participants in the user study were 200 individuals with psychiatric sequelae, recruited through a crowdsourcing platform. They were exposed to one of the four recommendation engines: Mozart, Haydn, Salieri, or a SOTA Visual baseline engine. The study involved assessing baseline affective states, eliciting music and painting preferences, facilitating a guided AT session with recommended artworks, and finally, assessing post-test affective states and collecting feedback on recommendation quality.
The results showed that all engines were rated similarly across various user-centric metrics of recommendation quality, such as accuracy, diversity, novelty, serendipity, immersion, and engagement. Importantly, a significant mood enhancement effect was observed across all groups after the guided art therapy session. Before the therapy, many participants reported being in a negative mood (46.6%), which shifted to a majority reporting a positive mood (72.6%) after the session. While the differences between the new music-based approaches and the visual baseline were not statistically significant in mood changes, the music-based methods demonstrated a competitive therapeutic profile.
A thematic analysis of user reflections revealed that all engines predominantly elicited feelings of ‘Hope,’ with Salieri showing the highest focus on this theme. This suggests that music-based preference elicitation methods are a viable and effective alternative for personalized art therapy. The findings indicate that these CDR algorithms unlock new possibilities by mapping music’s emotional cues to therapeutic visual art recommendations, potentially allowing future AT systems to integrate with music streaming applications to leverage user listening history for personalized recommendations.
Also Read:
- Unlocking Deeper Connections: AI-Powered Movie Recommendations Based on Emotional Tone
- Crafting Dynamic Soundtracks: A New AI Framework for Video-to-Music Creation
While promising, the research acknowledges limitations, primarily the scarcity of therapeutically curated painting collections with reliable affective labels. Future work could focus on crowdsourced studies to build more robust therapeutic datasets. This pioneering work advances recommender systems by addressing single-domain limitations and aims to inspire further multimodal recommender systems for well-being. You can find the full research paper here: Affect-aware Cross-Domain Recommendation for Art Therapy via Music Preference Elicitation.
