Poster A66, Saturday, March 25, 5:00 – 7:00 pm, Pacific Concourse
Influence of Speakers’ Gaze on Listeners’ Comprehension: Evidence from Event Related Potentials (ERP)
Torsten Kai Jachmann1, Heiner Drenhaus1, Maria Staudte1, Matthew W. Crocker1; 1Saarland University, Germany
Eye-tracking studies have shown that language external cues such as gaze influence listeners’ visual attention and affect sentence comprehension (Staudte, 2011). However, little is known about the underlying nature of processing difficulties of contradictory visual and auditory information. We present findings from an ERP study (30 German right-handed participants (18–32)) that utilized a stylized face performing gaze cues time-aligned to an auditory sentence. The sentence described a size or brightness comparison between two out of three objects present in a visual scene (fully counter-balanced). Gaze cues preceded the naming of the object by 800ms in order to obtain a natural gaze behavior (Kreysa, 2009). The gaze cue toward the second named object was manipulated such that it was either congruent (toward the object), incongruent (toward the unnamed object), or neutral (toward the bottom of the screen). ERPs time-locked to the onset of the noun following the manipulated gaze cue revealed that, compared to the congruent condition, incongruent and neutral gaze cues evoked an early-starting posterior negativity (150ms – 450ms). The incongruent gaze cue additionally induced a late sustained posterior positivity starting around 500ms after noun onset (500ms – 1000ms). The results suggest that congruent gaze facilitates comprehension, as manifest by a reduced negativity (N400), whereas incongruent gaze not only interferes with lexical retrieval of the noun, as revealed by modulation of the N400 (e.g.; Kutas, 2011), but further entails an update of the situation build on the preceding visual information, as indexed by the late positivity (e.g.; Polich, 2009).
Topic Area: LANGUAGE: Semantic