Schedule of Events | Search Abstracts | Invited Symposia | Symposia | Poster Sessions | Data Blitz

Sketchpad Series 

Extracting visual and linguistic responses to continuous natural sign language

Poster Session D - Monday, March 9, 2026, 8:00 – 10:00 am PDT, Fairview/Kitsilano Ballroom

Aaron Nidiffer¹, Chen Guo¹, Matthew Dye², Edmund Lalor¹; ¹University of Rochester, ²Rochester Institue of Technology

While we communicate, our brains often must parse and process a continuous barrage of sensory information in real time. Although a lot is left to be discovered, the study of natural connected speech has yielded vast insight into how the brain receives, encodes, and transforms acoustic information into neural representations that carry meaning. We have also gained tremendous insight into how these processes are affected by other sensory inputs or cognitive processes like attention and predictive processing. Yet we cannot be sure whether the knowledge is general or relevant only to speech. Studies of natural sign language are yet few and the field is ripe for study. To understand signed language more specifically and natural language more broadly, it is imperative to develop our current tools to fit the structure of other languages. In this presentation, we will highlight relevant similarities and differences between speech and sign and discuss their implications on how we analyze neural data recorded during sign. We will also present preliminary analysis of electroencephalography (EEG) recorded while fluent signers watched a short story in American Sign Language (ASL). ASL videos will be analyzed in terms of their low-level physical structure, such as luminance changes or sign pose estimation, and higher-level structures associated with linguistic content and meaning. Temporal response functions will be fit to EEG using these aforementioned stimulus features derived from the ASL videos. Finally, we’ll discuss some caveats, benefits, and potential fruitful future directions in the study of sign language processing.

Topic Area: LANGUAGE: Other