Can ChatGPT Be Useful for Distant Reading of Music Similarity

We explore whether large language models (ChatGPT) can be used as a ‘distant reading’ tool to estimate music similarity between songs from textual information, complementing experiments with human participants. We compare degrees of rater agreement to previous results from a listening test, showing that correlation of ChatGPT with human raters is significantly lower than the average human inter-rater agreement, but nevertheless still of moderate positive size. We discuss whether an approach based on the largely opaque ChatGPT model can be scientifically valid and to what extent it allows transparent evaluation of music information research experiments.