Beyond Object recognition : The Role of Visual-Semantic Representations in Understanding the Ventral Visual Stream

Poster Session F - Tuesday, April 16, 2024, 8:00 – 10:00 am EDT, Sheraton Hall ABC

Siddharth Suresh¹ (siddharth.suresh@wisc.edu), Kushin Mukherjee¹, Timothy Rogers¹; ¹University of Wisconsin-Madison

In visual cognitive neuroscience, there are two main theories about the function of the ventral visual processing stream (VVS). One suggests that it serves to classify objects (classification hypothesis) by learning categorical representations, which can then be indexed into the human semantic system to allow for the retrieval of other information; the other suggests that the VVS generates intermediate distributed representations from which people can generate verbal descriptions, actions, and other kinds of information (distributed semantic hypothesis). The classification hypothesis suggests that the visual system is isolated from the brain’s semantic system in terms of computations, hence Deep Convolutional Neural Networks trained on categorization seem to be good models of the VVS. However, the distributed semantic hypothesis suggests that the visual system is integrated with other systems such as language, praxis, etc. To adjudicate between these competing hypotheses, we trained two deep convolutional AlexNet models on 330,000 images belonging to 86 categories, representing the intersection of Ecoset images and the semantic norms collected by the Leuven group. One model was trained to produce category labels(classification hypothesis), and the other to generate all of an item’s semantic features (distributed semantic hypothesis). The two models learned very different representational geometries throughout the network. We also estimated the human semantic structure of the 86 classes by using a triadic comparison task. The representations acquired by the feature-generating model aligned better with human-perceived similarities amongst images, and better predicted human judgments in a triadic comparison task. The results thus support the distributed semantic hypothesis.

Topic Area: PERCEPTION & ACTION: Vision