TLDR: A new research paper introduces a method to improve music recommendation systems by balancing accuracy with novelty. It uses “sub-item” popularity (sPPS) alongside traditional “item-level” popularity (PPS) to understand user preferences at a finer detail. This allows the system to recommend both familiar tracks and new, relevant music, significantly boosting novelty without losing accuracy, leading to better user engagement.
In the evolving world of music recommendation, systems aim to predict what users will want to listen to next. A common challenge in this domain is repetitive listening, where users frequently replay their favorite tracks. While current sequential recommendation models, often based on advanced Transformer architectures, are good at understanding user preferences over time, they sometimes struggle with this repetitive behavior. To address this, Personalised Popularity Scores (PPS) were introduced, which quantify how often a user listens to specific tracks, significantly improving prediction accuracy for repeated content.
However, relying too heavily on PPS can lead to a different problem: the system might keep recommending content the user already knows, limiting their exposure to new or unexpected music. This reduces “serendipity,” which is crucial for keeping users engaged and satisfied in the long run. To tackle this, new research builds upon a framework called RecJPQ, which was originally designed to handle very large music catalogs by breaking down items into smaller “sub-items.”
The researchers repurposed RecJPQ’s sub-item architecture to model personalized popularity at a much finer level. Instead of just looking at whole songs, they analyze “sub-IDs” – smaller components that represent shared characteristics like genre or artist. By tracking how often these sub-IDs appear in a user’s history, the system can identify implicit preferences for underlying musical attributes. This new approach, called sub-ID-level Personalised Popularity Score (sPPS), allows for a more nuanced understanding of user preferences, capturing patterns that aren’t visible when only considering entire songs.
The core idea is to combine two types of personalized popularity signals: Item-ID Popularity (the original PPS, measuring how often a user listens to a specific track) and Sub-ID Popularity (sPPS, leveraging RecJPQ’s sub-IDs to capture latent repetition and shared components across items). These two signals are integrated with the output of the main recommendation model using a weighted scoring function. This flexible approach means it can be applied to various sequential recommender systems without needing to redesign the entire model.
Experiments were conducted on two music streaming datasets, Yandex and Last.fm-1K, using a state-of-the-art Transformer-based model called BERT4Rec, integrated with RecJPQ. The goal was to see how these different popularity signals affect the balance between recommendation accuracy (how well the system predicts what a user will like) and personalized novelty (how often it recommends something new and relevant).
The findings showed that while item-level PPS significantly improves accuracy, it often comes at the cost of reduced novelty, reinforcing content the user has already heard. In contrast, sub-ID popularity (sPPS) consistently outperformed item-level PPS in terms of personalized novelty. It achieved higher novelty scores without sacrificing recommendation accuracy. This suggests that by understanding structural recurrence at the sub-item level, the system can uncover hidden user preferences and recommend more diverse content.
When both signals were combined, the system showed a robust foundation from the sub-ID signal, allowing the item-level PPS to further improve accuracy with minimal impact on novelty. This demonstrates that the new method provides explicit control over the trade-off between accuracy and personalized novelty, helping the system suggest unexpected yet relevant items.
Also Read:
- New AI Breakthrough: Estimating Musical Surprisal from Audio
- Breaking the Filter Bubble: How AI Uncovers Hidden Interests for Better Recommendations
This work highlights the significant benefits of integrating personalized popularity signals into music recommender systems. By extending the RecJPQ framework with both item-ID and sub-ID level popularity modeling, the researchers have enabled fine-grained personalization while maintaining efficiency. This approach offers a powerful way to balance predicting what users already love with introducing them to new music they might enjoy, fostering long-term engagement and satisfaction. You can find more details about this research in the full paper available here.
