Conference on Cognitive Computational Neuroscience

To navigate the immediate visual environment, humans use a variety of locomotive actions, such as walking, swimming or climbing. How does the brain represent such environmental action affordances and which visual features drive these representations? Here, we compared representations of visual properties derived from human annotations, fMRI measurements, and convolutional neural networks (CNNs) on a new set of real-world scenes that afford distinct locomotive actions in a diverse set of indoor and outdoor environments. Representational similarity analysis shows that scene-selective brain regions represent information about action affordances as well as materials and objects. In contrast, CNNs trained on scene classification show comparatively lower correlation with action affordances, instead most strongly representing global scene properties. Together, these results suggest that specialized models that incorporate action affordances may be needed to fully capture representations in scene-selective visual cortex.