Natural action representations in the mind and brain

Poster Session F - Tuesday, April 16, 2024, 8:00 – 10:00 am EDT, Sheraton Hall ABC

Diana C Dima^1,2 (ddima@uwo.ca), Jody C Culham¹, Yalda Mohsenzadeh^1,2; ¹Western University, ²Vector Institute for Artificial Intelligence

Humans rapidly extract information about other people’s actions in many natural settings. How does the brain process this complex information, from perceptual details to abstract concepts? To address this, we curated a naturalistic dataset of 95 short videos and sentences depicting everyday human actions. We then labeled each action with four semantic features categorizing actions at different levels of abstraction, ranging from action verbs (e.g., chopping) to broad action classes (e.g., manipulation) and the actions’ target (e.g., an object). We also annotated the stimuli with other relevant perceptual and action-specific features. In two behavioral experiments, participants arranged the videos and sentences according to their similarity in meaning. The broadest semantic features, particularly the actions’ target, explained the most unique variance in behavioral similarity judgments across vision and language. We next collected temporally and spatially resolved neural data (EEG and fMRI) while participants viewed the videos and sentences. We found that actions were processed along a temporal gradient in the brain, from perceptual features to semantic information about the actions’ target and class. We mapped these semantic features to areas in lateral occipitotemporal cortex. Finally, using cross-decoding across videos and sentences, we identified a late (500 ms) modality-invariant neural response. Together, our results characterize the computations underlying natural action understanding in the mind and brain, and highlight the shared representations of human actions across vision and language.

Topic Area: PERCEPTION & ACTION: Vision