Poster B59, Sunday, March 25, 8:00-10:00 am, Exhibit Hall C
Modeling the dynamic content, encoding, and retrieval of naturalistic stimuli
Andrew Heusser1, Jeremy Manning1; 1Dartmouth College
The dynamic content of naturalistic stimuli is much more richly structured than traditional (highly controlled, but impoverished) stimuli, interweaving many simultaneous interacting information streams. This presents a substantial challenge to studying naturalistic cognitive phenomena. For example, to study how participants encode and retrieve information contained in a video, one needs to formally define (a) the dynamic informational content of the video (b) a means of assessing the participant's memory, and (c) a means of matching up the participant's responses with specific moments of the viewed video. We present a methodological advance for studying naturalistic learning and memory. Specifically, we develop an automated pipeline that uses machine learning algorithms to extracting a text description of each frame of video and moment of audio. We then apply topic models (Blei et al., 2003) to the extracted text (treating each moment of video as a "document"). This yields topic vectors (i.e. a mix of themes) for each moment of video. We apply the same pipeline to spoken responses from participants, and we use the match between the topic vectors of the video and participant's responses as an indication of which moments of video each response is about. We apply our approach to data collected as participants viewed and verbally recalled an episode of Sherlock (Chen et al., 2017). Our automated approach replicates behavioral results from the original study that previously required manually matching up each moment of video and response.
Topic Area: LONG-TERM MEMORY: Episodic