Neural decoding of semantic representations from novice sign language learners reflects newly-acquired vocabulary

Poster Session B - Sunday, April 14, 2024, 8:00 – 10:00 am EDT, Sheraton Hall ABC

Megan E. Hillis¹ (megan.e.hillis.gr@dartmouth.edu), Brianna Aubrey¹, Julien Blanchet¹, Qijia Shao², Xia Zhou², Devin Balkcom¹, David J. M. Kraemer¹; ¹Dartmouth College, ²Columbia University

How is newly-learned information reflected in the brain? Prior work has found that data-driven neuroimaging methods such as multivariate representational similarity analysis (RSA) can be used to characterize the emergence of newly-learned concepts over the course of learning in a number of domains. Furthermore, studies of semantic processing suggest that the same concept presented in two different modalities, such as homologous words in two different languages which participants speak fluently, can evoke similar neural patterns associated with underlying semantic meaning. In the present study, forty hearing English speakers with no prior experience in American Sign Language (ASL) completed a series of short online learning modules followed by an fMRI scan during which they viewed video clips of both previously-studied and new, unstudied words in ASL. Using multivariate pattern analysis methods including RSA and support vector machine (SVM) classification, we identified brain regions that were reflective of categorical distinctions between the stimuli in English which also tracked semantic relationships between the studied (but not unstudied) ASL words at the group level. We further investigate the ability of individual-level neural patterns to predict student performance on a recall quiz immediately before the scan as well as one week later. Our results provide evidence for the ability of multivariate neuroimaging analysis approaches to detect shifts in understanding even in the earliest stages of language learning.

Topic Area: LANGUAGE: Semantic