Contrastive Decoding for Concepts in the Brain

Abstract

This thesis presents a novel data-driven approach for identifying categoryselective regions in the human brain that are consistent across multiple participants. By leveraging a massive fMRI dataset and a multi-modal (language and image) neural network (CLIP), we trained a highly accurate contrastive brain decoder to predict neural responses to naturalistic images in the human visual cortex. We then applied a novel adaptation of the DBSCAN clustering algorithm to identify clusters of voxels across multiple brains that decode similar concepts, which we term shared decodable concepts (SDCs). The SDCs are interpreted by identifying the closest embeddings to each cluster centroid and analyzing the associated images and text. In contrast to other methods, ours does not require registration to a template space, allowing us to maintain the unique functional layout of each participants brain. It also uncovers both activating and deactivating stimuli, highlighting the importance of both in understanding brain function. Our approach allowed us to uncover categoryselective areas for food, subcategories of bodies and places, color, numerosity, object size, softness, lighting conditions, and more, demonstrating the versatility and potential of our approach for exploring brain functions.