Poster Session D, Monday, March 25, 8:00 – 10:00 am, Pacific Concourse
Deep learning classifiers of visual cortex activity can identify which moment of a video is represented by a single fMRI volume during naturalistic movie viewing
Matthew Johnson1, Jacob Williams1, Rafay Khan2, Karl Kuntzelman1; 1University of Nebraska-Lincoln, 2University of Illinois at Urbana-Champaign
In this fMRI study, participants watched the same short video several times in separate scan runs. We then trained subject-specific deep learning models based on a visual cortex region of interest (using the DeLINEATE toolbox; http://delineate.it) to discriminate whether two fMRI volumes from different movie-viewing runs represented the same timepoint vs. two different timepoints. All training was performed only on data from the second half of each run. The trained deep-learning model was then applied to the held-out, independent first half of each run, comparing each timepoint of each run to each timepoint of each other run. We found that this model performed easily above chance at identifying whether two timepoints were the same or different. Furthermore, the likelihood of “same” judgments fell off with increasing temporal distance between the timepoints compared. This suggests that such classifiers can provide a useful and robust measure of neural pattern similarity, even for continuously varying brain activity without discrete epochs or events. Our method compared favorably to other similarity measures (Pearson correlation, Euclidean distance) while avoiding certain caveats of those measures and yielding a more interpretable numeric value. In a control analysis, we also compared each timepoint of each run of movie-viewing to each timepoint of each run of two resting-state scans. The control analysis did not find strong similarity between corresponding timepoints, confirming that effects observed among movie-viewing runs were due to sensory/cognitive activity evoked by the film, rather than alternative explanations such as scanner drift or time-on-task.
Topic Area: METHODS: Neuroimaging