Schedule of Events | Search Abstracts | Invited Symposia | Symposia | Poster Sessions | Data Blitz

NeuroVLM: A Bi-Directional Vision-Language Framework Linking Brain Activation Maps and Cognitive Functions

Poster Session A - Saturday, March 7, 2026, 3:00 – 5:00 pm PST, Fairview/Kitsilano Ballroom
Also presenting in Data Blitz Session 4 - Saturday, March 7, 2026, 10:30 am – 12:00 pm PST, Salon F.

Borngreat Omoma-Edosa¹, Ryan Hammonds², Jerjes Aguirre Chavez³, Bradley Voytek⁴; ¹University of California San Diego

Human neuroimaging studies have generated vast multimodal datasets pairing textual descriptions with spatial brain activation coordinates, yet existing approaches remain limited in bridging these modalities. We introduce NeuroVLM, a bi-directional, generative vision-language framework that unifies neuroimaging and natural language to enable both text-to-brain and brain-to-text decoding for arbitrary (non-dictionary-based) natural language text or brain images. Our NeuroVLM model integrates 27,000 coordinate-based activation maps, 27,000 neuroscience publications and 3,000 common neuroscience articles into a shared latent space. This extension implements the inverse pathway, translating neuroimages into interpretable textual, cognitive, and functional outputs. Our architecture combines a fine-tuned transformer-based language encoder with a novel 3D neuro-autoencoder that compresses activation likelihood maps into 768-dimensional embeddings. Both modalities are aligned through contrastive and reconstruction objectives, allowing accurate retrieval, decoding, and similarity-based reasoning. Training leveraged large open-access repositories including PubMed Central and Neurosynth, with ten-fold cross-validation assessing reconstruction fidelity (MSE, SSIM, Dice) and semantic retrieval. Results show that NeuroVLM-Inverse achieves decoding and ranking performance comparable to large-scale models while maintaining greater computational efficiency. Beyond methodological innovation, the system enables lesion-to-function inference, cognitive-state classification, network labeling, and a neuroscience-specific search engine linking patient scans to literature. By bridging natural language, brain representations, and large-scale data, NeuroVLM-Inverse advances toward interpretable, accessible, and clinically relevant brain-language alignment, offering a step toward the next generation of cognitive and computational neuroscience tools.

Topic Area: METHODS: Neuroimaging