Conference on Cognitive Computational Neuroscience

Humans and animals have a rich and flexible understanding of the physical world, which enables them to infer the latent states of objects and events, plausible future states, and use that to plan and anticipate the consequences of actions. However, the neural mechanisms underlying these computations are unclear. Here we take a goal-driven modeling approach to this question by constructing a large class of neural networks to predict the future state of rich, ethologically-relevant environments. By testing these models on their ability to generalize to a novel ball interception task and comparing their internal units to neural responses in frontal cortex while primates perform this task, we identify key principles of mental simulation. Specifically, we find that neural dynamics are currently best predicted by models trained to predict the future state of their environment in a latent state optimized for dynamic scenes in a temporally contrastive, self-supervised manner. These models also approach the neurons' ability to predict the ground truth ball position while it is occluded, despite not being explicitly trained to do so. Overall, these findings suggest that frontal cortex is thus far consistent with being optimized to future predict on dynamic, scene-centric representations.