Conference on Cognitive Computational Neuroscience

Incorporating uncertainty derived from category judgments has been instrumental in improving cognitive models of categorization and natural image classification. How this uncertainty is measured, and when and where different types of uncertainty are most useful, remain largely unexplored questions. We investigate the performance of a range of natural image classifiers trained on different types of human uncertainty, expressed as “soft labels” over image categories. We find that training with (the right kind of) softer labels allows these networks to preserve their ability to generalize performance under conditions of increasing distributional shift. We explain this in terms of better representation learning—softer labels that better capture the similarity structure inherent in natural images confer more robust representational information.