Poster Session E, Monday, March 25, 2:30 – 4:30 pm, Pacific Concourse
An Independent Component Analysis Approach to Assessing the Integration of Faces and Voices in Multimodal Emotion Perception: An Electroencephalography Study
Katherine Becker1, Donald Rojas1; 1Colorado State University
Emotion is vital to communication as it instantaneously connects us through subtle changes in facial movements and vocal expressions, with perception occurring via the simultaneous integration of affective vocal and facial information. These expressions activate a constellation of brain areas sensitive to emotional facial expressions, which are distinct from those devoted to prosody recognition. While much is known about the independent contributions of these channels to emotion perception, less research has focused on multimodal affect perception. This study sought to quantify the cognitive and neural underpinnings of this dynamic process by extracting independent components from 39 electrodes. Brain activity was measured using an EEG while subjects (n=26) were presented with angry or happy faces (FO), voices (VO), or faces and voices (F+V) presented together. Prosodic stimuli consisted of vocalizations (/a/) produced in neutral, angry, and happy tones. ICA components were extracted using group ICA and the trial-wise component waveforms were correlated with the task structure at each time point. All task conditions were significantly correlated with neural activity, corrected for multiple comparisons FDR q < .05. FO and F+V exhibited a N170 like ERP over parieto-occipital electrodes. VO exhibited a PSW potential over frontotemporal sites. Activity was positively correlated with F+V and negatively correlated with both unimodal conditions (200-800ms). Component activity was correlated with anger and happiness, which exhibited anti-correlated activity (250-350, 350-900ms). These results indicate that the brain exhibits a unique pattern of activity when processing multimodal emotional stimuli, which is distinct from that witnessed in unimodal processing.
Topic Area: EMOTION & SOCIAL: Other