Seeing Speech in a New Light: Augmenting Speech Performance using Lip Movement with Imperceptible Light

Poster Session F - Tuesday, April 16, 2024, 8:00 – 10:00 am EDT, Sheraton Hall ABC

Hyojin Park¹ (h.park@bham.ac.uk); ¹University of Birmingham

In challenging listening conditions, speech comprehension can be improved through the utilization of visual cues, particularly those related to lip movement. However, can speech comprehension be further enhanced by adopting a proactive approach, such as employing non-invasive rhythmic stimulations over the sensory areas and inducing their intermodulation effects? This study aimed to investigate the causal and modulatory effect of audiovisual speech integration in challenging listening conditions by employing the rapid (yet imperceptible) frequency tagging (RIFT) technique with low-frequency amplitude modulation, measured through magnetoencephalography (MEG). The current study employed a naturalistic audiovisual speech paradigm in conjunction with a dichotic listening task, where participants directed their attention towards one of two speakers while ignoring the other. The lip movements of the speaker were tagged at 55 Hz while the auditory speech was tagged at 40 Hz. Additionally, to examine the impact of rhythmic modulation on speech comprehension performance, the visual tagging frequency was modulated by the low-frequency amplitude derived from either the attended or unattended speech while brain activities were captured using MEG. The results revealed significant effects of both auditory and visual tagging in their respective sensory cortices across all experimental conditions. Moreover, higher-order brain regions exhibited representations of intermodulation frequencies in the right temporo-parietal junction and superior/middle temporal gyrus, specifically in the condition where the visual tagging was amplitude-modulated by the attended speech. These findings highlight the potential of non-invasive sensory stimulation through RIFT as a promising tool for enhancing speech intelligibility, particularly in environments where multiple speakers are present.

Topic Area: ATTENTION: Multisensory