TLDR: A research paper introduces a manually labeled dataset of pop song lyrics with emotional intensity scores for six basic emotions (joy, sadness, anger, fear, surprise, disgust), collected using a Mean Opinion Score (MOS) approach. It evaluates both fine-tuned BERT-based models and a zero-shot Grok 3 LLM, finding that fine-tuned models significantly outperform the zero-shot approach in accurately predicting multi-label emotion scores in lyrics.
Understanding the emotional core of song lyrics is crucial for how listeners connect with music and what they choose to listen to. A new research paper, “From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics,” delves into this complex area, aiming to accurately predict emotional intensity in song lyrics. The study, conducted by Shay Dahary, Avi Edana, Alexander Apartsin, and Yehudit Aperstein, explores how well different language models can identify and quantify emotions like joy, sadness, anger, fear, surprise, and disgust in popular songs.
The researchers highlight that while musical elements contribute significantly to emotional perception, the lyrical content often provides the most direct and interpretable emotional cues. Accurately understanding these cues can lead to better personalized music recommendations, emotionally-driven playlist generation, and even applications in mental health interventions or cultural studies. However, automatically attributing emotions to lyrics remains a challenging problem due to abstract language, metaphors, and cultural references. Traditional sentiment analysis methods, typically designed for binary or ternary sentiment classification, often fall short in capturing the rich, multi-dimensional, and overlapping emotional content present in lyrical texts.
Building a Reliable Emotional Dataset
To tackle these challenges, the team constructed a high-quality, manually labeled dataset. They employed a method called Mean Opinion Score (MOS), where a diverse committee of human annotators, each with prior experience in affective text analysis or music interpretation, participated in the labeling process. Annotators were asked to read each song lyric and independently assign an intensity score ranging from 0 (not present) to 5 (highly present) for each of Ekman’s six basic emotions: joy, sadness, anger, fear, surprise, and disgust.
For each lyric and each emotion category, the individual scores were aggregated by computing the arithmetic mean across all annotators, resulting in continuous-valued emotional intensity scores. These MOS values effectively captured the subjective variability among annotators while providing a consistent numerical representation of emotional attribution. This approach also allowed the dataset to reflect the degree of emotional presence rather than forcing binary or categorical labels, which are often inadequate for representing the nuanced emotional content found in song lyrics. In the resulting dataset, the MOS scores for all emotions were clustered near mid-range values, with joy having the highest mean score (0.95) and anger the lowest (0.40). Interestingly, the study found no significant correlation between music genre and emotion scores, with the notable exception of anger being more prominent in rap songs.
Evaluating Language Models
Using the constructed dataset, the researchers evaluated both fine-tuned and zero-shot models for predicting multi-label emotion scores.
First, Fine-Tuned BERT-Based Models: The team fine-tuned a pre-trained BERT model, and also a RoBERTa model, for regression-based multi-label prediction. These models were designed to directly predict the six continuous emotion scores for each lyric. The model architecture consisted of the standard BERT encoder followed by a fully connected regression head with six output nodes, one for each emotion. The loss function used was Mean Squared Error (MSE), which is appropriate for continuous-valued target variables. These models were trained and validated on the annotated dataset to optimize their predictive accuracy across all six emotions.
Second, Zero-Shot Pretrained Models: In addition to fine-tuned models, the study investigated the zero-shot emotion classification capabilities of large language models without requiring further task-specific training. They used the Grok 3 model for this evaluation. For zero-shot evaluation, each model was prompted using carefully crafted instructions to predict the emotional intensity scores directly from the raw lyrics. The models were asked to output numeric scores corresponding to the six target emotions. This experimental setup enabled the researchers to evaluate the generalization capabilities of large pre-trained models and their ability to perform fine-grained emotional analysis without requiring specialized fine-tuning.
Key Findings: Fine-Tuning Leads the Way
The experimental results clearly demonstrated that the fine-tuned models (uncased BERT and RoBERTa) achieved substantially lower Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) values compared to the zero-shot Grok LLM. This underscores the effectiveness of task-specific training on the manually annotated dataset. The performance gap was especially evident in the aggregated metrics, where fine-tuned models reduced error rates by more than 30% relative to the zero-shot approach. A more detailed breakdown highlighted that this improvement was consistent across all six basic emotions.
Notably, the fine-tuned models demonstrated strong predictive accuracy for emotions such as surprise and fear, with MAE values as low as 0.10 and 0.13, respectively. By contrast, the zero-shot model exhibited higher error rates across all categories, particularly for joy and anger. These results confirm that while zero-shot LLMs provide a baseline capability for lyric emotion attribution, fine-tuned transformer-based models offer far superior performance, making them more suitable for music information retrieval and affective computing applications.
Also Read:
- Assessing Topic Model Quality with Large Language Models: A New Framework
- Unpacking LLM Effectiveness: Fine-Tuning Outperforms In-Context Learning for Misinformation Detection
Looking Ahead
In conclusion, this study addressed the complex task of emotional attribution in song lyrics by constructing a high-quality, manually labeled dataset using the Mean Opinion Score (MOS) method. This approach enabled the researchers to capture the nuanced and subjective emotional content present in lyrics across Ekman’s six basic emotions. The results demonstrate that fine-tuning a BERT-based model significantly outperforms a large zero-shot model such as Grok 3. While zero-shot models offer a practical solution when labeled data is unavailable, their performance remains inferior to models explicitly trained for the task.
Future work will explore expanding the dataset to include a broader range of lyrical genres and languages, which may improve model generalization and robustness. Additionally, the researchers plan to investigate the effectiveness of advanced prompting strategies, such as chain-of-thought reasoning and structured output prompts, to enhance zero-shot performance. Ultimately, the aim is to investigate the application of larger and more specialized language models, as well as multimodal approaches that integrate both lyrics and audio features, to further advance the field of emotion recognition in music. You can find the full research paper here: From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics.
