Schedule of Events | Search Abstracts | Invited Symposia | Symposia | Poster Sessions | Data Blitz

Modality-Specific Influences in Audio-visual Emotion Perception: Valence and Arousal Representations

Poster Session B - Sunday, March 8, 2026, 8:00 – 10:00 am PDT, Fairview/Kitsilano Ballroom

Yongseong Lee¹ (), Youngju Lee¹, Dongha Lee; ¹Korea Brain Research Institute

Prior research found a multimodal gain in audio-visual emotion recognition, highlighting the importance of real-world contexts in emotion studies. However, it remains unclear how visual and auditory modalities contribute to valence and arousal in emotion perception, as well as whether they constitute distinct affective representational patterns across modalities. We hypothesized that the visual modality is more dominant in emotion perception than the auditory modality, particularly for positive emotions. To test the hypothesis, we conducted an emotion perception task using audio-only (AO), video-only (VO), and audio-visual (AV) speech stimuli, in which valence and arousal were measured. Additionally, we conducted behavior-based representational similarity analysis (RSA) to investigate whether different sensory modalities exhibit distinct combined valence–arousal representational patterns. We found that the AO differed from the other modalities. In particular, RSA revealed distinct representational structures across modalities, indicating that the AO exhibited a markedly different affective similarity structure from VO and AV. For the individual emotions, “happy” and “disgusted” showed that AV patterns were more similar to VO patterns than to AO patterns, indicating greater influence of the visual modality. The “surprised” emotion had a neutral valence overall, suggesting that opposing valence cues—positive from AO and negative from VO—were integrated, resulting in neutrality. These findings indicate that sensory modalities differ not only in their impact on mean valence and arousal but also in the structure of affective representations, suggesting that individuals use adaptable perceptual strategies according to the specific features of each emotion.

Topic Area: EMOTION & SOCIAL: Person perception