TLDR: The MusGO (Music-Generative Open AI) framework is a new community-driven tool designed to assess the openness of music-generative AI models. Developed by adapting an existing framework for LLMs and refined with feedback from the Music Information Retrieval (MIR) community, MusGO defines 13 categories of openness (8 essential, 5 desirable). It was used to evaluate 16 state-of-the-art models, revealing significant variations in transparency, particularly regarding training data. The framework aims to promote transparency, accountability, and responsible development in music AI, while acknowledging challenges like intellectual property rights and the need to consider broader ethical implications beyond just openness.
Generative Artificial Intelligence (AI) has made rapid strides in the music domain since 2023, offering exciting new possibilities for creation. However, this advancement also brings significant ethical challenges, particularly concerning transparency, accountability, and the potential replication of artists’ work. As regulations like the EU AI Act encourage open models, there’s a growing debate about what truly defines an ‘open’ AI model.
To address this crucial need for clarity, researchers have developed MusGO (Music-Generative Open AI), a new community-driven framework designed to assess openness specifically in music-generative AI. This framework adapts an existing evidence-based method used for Large Language Models (LLMs) and tailors it to the unique complexities of music.
Developing the MusGO Framework
The creation of MusGO involved a collaborative approach, actively engaging the Music Information Retrieval (MIR) community. An anonymous online survey was conducted with 110 participants, primarily academics from Europe and North America, to gather feedback on initial openness categories. Participants rated the relevance and their agreement with proposed statements for each category, providing valuable insights and suggestions for refinement.
Based on this community feedback, the framework was refined to include music-specific nuances. This involved clarifying category descriptions, defining what constitutes ‘partial’ and ‘fully open’ for each, and addressing challenges like sharing IP-protected training data. New categories were also introduced, such as ‘Evaluation procedure’ to cover model performance details, and ‘Supplementary material page’ to acknowledge the importance of dedicated websites with usage instructions and sonified examples, which are crucial in music research.
MusGO’s Structure and Assessment
The final MusGO framework comprises 13 openness categories, divided into two levels: 8 essential categories and 5 desirable categories. Essential categories, considered key for assessing openness, are graded on a three-level scale: closed, partial, and fully open. Desirable categories, which enhance openness but are not indispensable, are binary – either they exist or they don’t.
To put MusGO into practice, the researchers evaluated 16 state-of-the-art music generation models, including well-known names like Jukebox, MusicGen, and Stable Audio Open. Each model was assessed against the framework’s requirements, with multiple authors reviewing evaluations to ensure consistency. The findings are presented in a public leaderboard, which orders models based on a weighted openness score, giving more importance to highly relevant essential categories like ‘Model Weights’, ‘Training Data’, and ‘Open Code’.
Also Read:
- Enhancing Musical Onset Detection in Maracatu Through Adaptive Transfer Learning
- Unlocking Piano Expression: A New Model for Continuous Sustain Pedal Analysis
Key Findings and Implications
The assessment revealed significant variations in openness across the models. The ‘Training procedure’ category was the most open, with most models providing detailed information. Conversely, ‘Training data’ was the most closed, with only one model achieving fully open status, largely due to intellectual property rights and legal constraints in the music domain. There was a clear correlation between models that provided full access to source code, model weights, comprehensive code documentation, and appropriate licensing.
While MusGO provides a robust tool for evaluating openness, the paper acknowledges that openness alone doesn’t guarantee ethical behavior. The framework can, however, help expose ethical risks, such as copyright concerns or biases in training data. For instance, it can highlight whether models address issues like memorization or cultural appropriation. The authors emphasize that openness should be balanced with responsible AI practices and a broader discussion about the ethical, social, and economic implications of AI on musicians and creative professionals.
The MusGO framework and its accompanying leaderboard are designed to be continuously updated, serving as a live, collaborative platform for the community to track model evolution and contribute to a more transparent and responsible development of music-generative AI. You can find more details about the framework and the leaderboard in the full research paper: MUSGO: A Community-Driven Framework for Assessing Openness in Music-Generative AI.
