Conference on Cognitive Computational Neuroscience

Naturalistic stimuli evoke complex neural responses that can differ across people in their spatial and temporal properties. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment into corresponding events using a Hidden Markov Model (assuming exact spatial correspondence). Here, we propose a hybrid model, the Hyper-HMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project from voxels to a reduced-dimension latent space, in which timecourses are segmented into corresponding temporal events. This approach allows tracking of each individual's mental trajectory through an event sequence, and also allows for alignment with other feature spaces such as stimulus content. Using an fMRI dataset of angular gyrus responses to lecture videos, we demonstrate that the Hyper-HMM can be used to map all participants and the semantic content of the videos into a common low-dimensional space, and that these mappings generalize to a held-out video.