Poster D76, Monday, March 27, 8:00 – 10:00 am, Pacific Concourse
Audiovisual speech intelligibility decays under adverse listening conditions
Jess R. Kerlin1, Antoine J. Shahin; 1UC Davis, Center for Mind and Brain
For billions of people worldwide, daily communication is primarily conducted through a face-to-face natural conversation. However, little is known of how speech intelligibility changes over the course of a naturally spoken sentence, nor how visual speech is utilized as a sentence progresses. In the current study, we sought to characterize how audiovisual speech understanding evolves over the course of a sentence under adverse listening conditions. We presented individuals with sentences selected from over 1000 unique audiovisual English sentences contained in the TCD-TIMIT audiovisual corpus. Sentences were presented in silence or with background babble, with or without matching video of the speaker’s lip-movement and participants were asked to report all of the words of the sentence they heard. We expected that transcription accuracy under noisy listening conditions would increase over the course of the sentence due to improved speaker segregation and enhanced contextual predictability. Instead, we participant’s transcription accuracy substantially decreased with increasing word position when presented with background babble noise. This decrease in intelligibility could not be attributed to informational masking by the content of the background babble, and is not fully accounted for by reductions in talker volume over the course of a natural sentence. The availability of lip movement information substantially increased intelligibility across the entire sentence, particularly for frontally articulated phonemes, however, it did not alter the rate of decline over word position. These results suggest that while lip reading improves phonetic representations, it does not prevent a bottleneck that degrades retention of later information in memory.
Topic Area: LANGUAGE: Other