Poster D95, Monday, March 26, 8:00-10:00 am, Exhibit Hall C
Is a round shape integrated with a /bouba/ sound? Enhanced neuronal signals at the intermodulation frequencies of congruent audio-visual stimuli
Hiu Mei Chow1, Brianna Leonardo1, Aleksandra Sabov1, Vivian Ciaramitaro1; 1University of Massachusetts Boston
Sound-shape crossmodal correspondence is the association between abstract shapes and seemingly unrelated sounds, such as associating a round shape with a /bouba/ sound and a spikey shape with a /kiki/ sound. Are the stimuli naturally paired in crossmodal correspondence integrated? We used steady state evoked potentials (SSEPs), to examine if and how audio-visual stimuli congruent with sound-shape correspondence are integrated. We focused on brain responses at intermodulation frequencies (IM), the sum and differences of frequencies at which the shapes and sounds were presented. Given that IM responses are often enhanced when parts of visual stimuli are integrated (e.g. Boremanse, Norcia & Rossion, 2013), we hypothesized that if corresponding crossmodal stimuli are integrated then IM responses should be enhanced for congruent compared to incongruent pairings. Participants viewed two shapes (one round, one spiky) flickering at different frequencies (5.45Hz, 7.5Hz) while performing a central fixation task (color change detection). Both shapes were presented under each auditory condition: no sound, /ba/ sound (3Hz), or /ki/ sound (3Hz). We compared SSEPs at IM frequencies for congruent (e.g. round shape with /ba/) and incongruent (e.g. spikey shape with /ba/) pairings, and found enhanced IM signals for congruent pairings in occipital electrodes. Surprisingly, we also found enhanced IM signals for incongruent pairings in central and anterior electrodes. These effects were specific to the type of IM (sum vs. difference) and the frequency of the shape. Our findings serve as a first demonstration of using frequency tagging to study multisensory integration and conflict monitoring in crossmodal correspondence.
Topic Area: PERCEPTION & ACTION: Multisensory