Conference on Cognitive Computational Neuroscience

Midbrain dopamine (DA) activity is implicated in reinforcement learning (RL). DA activity is observed to signal reward prediction error (RPE), a key quantity in many RL algorithms used to modulate updates to state-reward and state-action associations. How the state representations underlying these associations are themselves learned is typically outside the scope of RL models of the basal ganglia. However, modern RL algorithms in machine learning typically employ deep RL, in which state representations are learned as part of the RL algorithm. In this work, we investigate the requirements for biologically plausible deep RL by studying a model in which DA projections to cortical regions upstream of the striatum drive state representation learning. We show that coarse, global DA projections are sufficient to enable effective representation learning. We next apply our model to an open question in the field: recent experimental evidence suggests DA population activity does not uniformly signal RPE, but rather heterogeneously encodes higher-dimensional information about an animal’s state. We show that such heterogeneity can improve learning by effectively providing auxiliary prediction objectives that refine representation learning. Our framework provides a normative prediction of the optimal set of DA signals for a given task.