Distinct Temporal Neural Dynamics between Audiovisual Perception and Cross-modal Mental Imagery

Poster Session D - Monday, April 15, 2024, 8:00 – 10:00 am EDT, Sheraton Hall ABC

Yu Hu¹ (yhu584@uwo.ca), Yalda Mohsenzadeh¹; ¹Western University

Mental imagery describes the process in which we internally create sensory experience without external input. Given most events in our daily life are multisensory, when we hear a sound we may find we are spontaneously imagining the associated visual in our mind. This type of mental imagery is triggered by the associated information from another sensory modality but remains less investigated. Previous studies found that the brain regions involved during perception and imagery are shared, but how the temporal dynamics would differ between the two distinct processes is still unknown. To address this, we conducted Electroencephalography (EEG) experiments on human participants (N=20) to compare the temporal dynamics between the perception of naturalistic audiovisual events in videos and the cross-modal imagery in which participants recalled the visual contents when hearing only the sounds as well as imagined the sounds when watching silent videos. With multivariate pattern analysis and temporal generalization method, we found that in contrast to the sequential processing of audiovisual perception, cross-modal mental imagery engages more sustained activity. We also observed that visual imagery involves a more delayed time course compared to auditory imagery, suggesting temporal disparities across different modalities of mental imagery. Overall, our study elucidates the temporal dynamics and distinctions in neural processing during audiovisual perception and cross-modal mental imagery.

Topic Area: PERCEPTION & ACTION: Multisensory