Modeling the implicit music representations in human brain with Deep Neural Networks

Abstract

Music processing is a fascinating and intricate phenomenon that has garnered significant research interest in recent years. Understanding how the human brain represents various music features is crucial for unraveling the mysteries of music perception and cognition. In this study, our objective was to investigate neural representations of music using functional magnetic resonance imaging (fMRI) to analyze the blood-oxygen-level-dependent (BOLD) signal activations in selected regions of interest (ROIs) during a continuous listening task. Additionally, we aimed to compare these neural activations with the hidden layer activations of a specific class of deep neural networks (DNNs). These DNNs, known as self-supervised models, have shown promise in capturing intricate patterns and encoding complex information. By utilizing representational similarity analysis (RSA), we aimed to explore the similarities and correlations between the neural representations of music features in the human brain and the hidden layers of the DNN encoder. Our findings revealed a correlation between the low-level music feature encoding observed in two important brain regions, namely the Superior Temporal Gyrus (STG) and Heschl’s gyrus (HG), and the hidden layers of the DNN encoder. This correlation provides evidence for the effectiveness of self-supervised DNNs as a reliable architecture for studying the domain of music processing. Importantly, this finding is particularly significant due to the limitations of naturalistic listening conditions in prior research studies. By bridging the gap between the neuroscientific investigation of music processing and the computational power of self-supervised DNNs, our study contributes to the growing body of research aiming to uncover the underlying mechanisms of music perception and representation. The implications of this research extend beyond the field of music, as the insights gained from studying music processing can potentially shed light on broader topics such as auditory cognition, neural encoding of complex stimuli, and the applications of deep learning in cognitive neuroscience