Highlight
Abstract letter concepts in the brain
Smolensky Highlight Fig 1
Achievement/Results
How do people intelligently cope with novel situations? A traditional answer is that, from extensive experience, we extract general, abstract concepts, and general knowledge interrelating these concepts. We recognize a novel situation as a particular instantiation of these general concepts, and apply our general knowledge to the new situation via these abstract concepts. Consider the abstract category a/A, which includes all instances of lower-case ’a’s and upper-case ’A’s, regardless of size, position, font, etc. If I see a funny image on a webpage asking me to type what I see as a security check, I succeed because, from experience with many letterforms, I have formed an abstract category of the letter a/A, and I recognize the leftmost bit of the image as an (unusual) instance of that category, so I hit the key that displays a familiar instance of the category a/A.
But another type of answer is that, as a byproduct of storing a lifetime’s experience with letters, I simply directly perceive the similarity of the keyboard’s ‘A’ to the funny shape on the screen: there is no part of my brain that encodes the abstract category a/A.
Until recently it was not imaginable that we could look inside the brain itself to see if it contains somewhere some structure that physically manifests the abstract category a/A. New results suggest that such structure does indeed exist.
In his doctoral research in the Cognitive Science Department of Johns Hopkins University, David Rothlein (Fig. 0) creatively integrated for the first time several state-of-the-art neural imaging methods in order to seek neural representations of abstract letter categories such as a/A. The interdisciplinary mix of methods he used was made possible by his training in the NSF-supported Integrated Graduate Education and Research Training (IGERT) program, ‘Unifying the Science of Language’.
In Rothlein’s experiment, participants undergoing functional Magnetic Resonance Imaging (MRI) were presented with images of single characters including lower- and upper-case letters. A hypothesis of neural network theory states that if the mind possesses a distinct representation of the abstract category a/A, then somewhere in the brain there should be a characteristic excitation pattern of a population of neurons that is triggered just when a stimulus is perceived as belonging to that category. The pattern of activity in this population when viewing an ‘A’ should be very similar to the pattern when viewing an ‘a’ and very dissimilar to the activation pattern in that population when viewing an ‘R’. This similarity structure is the signature of an abstract-letter-identity representation. It is different from the signature of a representation of pure visual form, divorced from letter identity: in such a representation, the activation patterns for ‘A’ and ‘R’ are more similar to each other than are the patterns for ‘A’ and ‘a’, because the visual form of ‘A’ is closer to that of ‘R’ than to ‘a’.
The use of such Representational Similarity Analysis (RSA) to identify signatures of different types of representation, developed by Kriegeskorte et al. in 2008, enabled comparison of the general types of information encoded in particular brain areas. To apply RSA to the relatively fine-grained question of abstract-letter-identity representation, Rothlein adapted it for relatively small pieces of brain tissue, cubes with 9-mm edges each containing 27 MRI voxels (image units). In such a ‘searchlight’ technique, roughly 8000 voxels in a large region are examined, one 27-voxel cube at a time. The value in each voxel roughly corresponds to how the activity in that voxel changes as a function of the letter stimulus that is presented. For each searchlight cube, the question is, do the activation patterns display the similarity signature of an abstract-letter-identity representation? Or the similarity signature of a visual form representation? Or that of several other representation types (e.g., the pronunciation of the letter’s name, its consonant-vowel status, or its case)?
Answering these questions required Rothlein to overcome a number of technical challenges. How to measure the similarity of activation patterns, to form the Representational Similarity Matrix (Fig. 1)? How to identify a statistically significant fit between this matrix and a similarity-pattern signature such as that of an abstract-letter-identity representation (Fig. 2)? Techniques for addressing such questions have been developed in the cognitive neuroscience literature, but adapting them to provide a coherent analysis of this novel experimental setting required a degree of mastery of both formal and experimental methods that is the particular goal of the IGERT program in which Rothlein trained.
The results of this analysis are shown in Fig. 2. Clusters of voxels (of significant size, shared by a significant number of participants) are colored by the representational signature that they matched. The signature for visual form representation (in green: ‘pixel-similarity’) was identified in the posterior occipital lobe—consistent with other findings concerning visual-spatial processing. The signature for representation of the pronunciation of the letter name (apparently rather automatic, as such naming was not called for in the experimental task) was identified within the bilateral posterior STS (near primary auditory cortex/Wernicke’s area) and the left inferior frontal gyrus (Broca’s area)— consistent with previous results on spoken-word processing. Critically, the abstract-letter-identity signature was identified within the left middle fusiform gyrus—a region associated with reading in prior studies. This is the only neuroimaging study that has demonstrated case-invariant representations in this region using single-letter stimuli.
These results bring us one step closer to resolving the long-standing debate over the existence of abstract knowledge in the human mind/brain.
Address Goals
The interdisciplinary mix of methods Rothlein used—combining crucial ideas from cognitive neuroscience, cognitive psychology, and mathematical methods of neural network theory—was made possible by his training in the Integrated Graduate Education and Research Training (IGERT) program, ‘Unifying the Science of Language’.
Conducting this novel research required Rothlein to overcome a number of technical challenges. How to measure the similarity of activation patterns, to form the Representational Similarity Matrix? How to evaluate the fit between this matrix and a similarity-pattern signature such as that of an abstract-letter-identity representation? How to determine whether a goodness-of-fit level is merely expected by chance, or truly significant? How to eliminate spurious voxels by exploiting the hypothesis that the voxels encoding a particular representation will be clustered together, rather than scattered all about the brain? How to correct for the fact that, when examining so many overlapping voxel-cubes, small clusters of apparently-significant matches to a representational signature will occur by chance? How to further eliminate accidental matches by looking for common representation locations across 9 participants with differing brain anatomy?
Techniques for addressing such questions have been developed in the cognitive neuroscience literature, but adapting them to provide a coherent analysis of this novel experimental setting required a degree of mastery of both formal and experimental methods that is the particular goal of the IGERT program in which Rothlein trained.
