c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 Available online at www.sciencedirect.com Journal homepage: www.elsevier.com/locate/cortex Note Schemas reveal spatial relations to a patient with simultanagnosia Alexander Kranjec a,b,*, Geena Ianni c and Anjan Chatterjee c a Psychology Department, Duquesne University, PA, USA Center for the Neural Basis of Cognition, Carnegie Mellon University, PA, USA c Center for Cognitive Neuroscience, University of Pennsylvania, PA, USA b article info abstract Article history: Maps, graphs, and diagrams use simplified graphic forms, like lines and blobs, to represent Received 23 August 2012 basic spatial relations, like boundaries and enclosures. A schema is an iconic representation Reviewed 3 October 2012 where perceptual detail has been abstracted away from reality in order to provide a more Revised 20 December 2012 flexible structure for cognition. Unlike truly symbolic representations of spatial relations Accepted 17 March 2013 (i.e., prepositions) a schema preserves some analog spatial qualities of the relation it stands Action editor Jason Mattingley in for. We tested the efficacy of schemas in facilitating the perception and comprehension Published online 27 March 2013 of spatial relations in a patient with bilateral occipitoparietal damage and resulting simultanagnosia. Patient E.E. performed six matching tasks involving WORDS (in, on, Keywords: above, below), photographic PICTURES of objects, and/or SCHEMAS depicting the same Simultanagnosia spatial relations. E.E. was instructed to match a single spatial relation to a corresponding Mental representation image from an array of four choices. On the two tasks that did not include matching to or Semantics from schemas, E.E. performed at chance levels. On tasks with schemas, performance was Spatial cognition significantly better, indicating that schematic representations make spatial relations visible in a manner that symbols and complex images do not. The results provide general insight as to how schemas facilitate spatial reasoning when used in graphic depictions, and how such theoretically intermediate representational structures could serve to link perceptual and verbal representations of spatial relations in the brain. ª 2013 Elsevier Ltd. All rights reserved. 1. Introduction Maps, graphs, and diagrams distill a more complex spatial reality into an easier-to-use model of the world by using simplified graphic forms, like lines and blobs, to represent basic relations, like boundaries and enclosures (Tversky et al., 2000). Graphical technologies have been used to convey culturally important information long before humans kept formal history. Prehistoric cave drawings use simplified visual elements like dots, lines, and rectangles to represent the abstract topologies and spatial arrangements of dwellings, paths, or crops (Chippindale and Nash, 2004; Smith, 1982). Pictograms were used for communicating important forms of cultural information well before the advent of symbolic writing systems (Gaur, 1985). Some forms of graphical representation (like calendars, for example) typically depict more * Corresponding author. Present address: Psychology Department, Duquesne University, 600 Forbes Avenue, 544 College Hall, Pittsburgh, PA 15282, USA. E-mail address: kranjeca@duq.edu (A. Kranjec). 0010-9452/$ e see front matter ª 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.cortex.2013.03.005 1984 c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 abstract concepts using a vocabulary of spatial relations. Such communication is partly accomplished by using spatial relations to represent more abstract meanings. For example, spatial length may represent temporal duration, and spatial contiguity may represent temporal sequence (Kranjec and Chatterjee, 2010). Thus, spatial semantics are particularly important for conveying both concrete and abstract meanings (Chatterjee, 2001; Talmy, 2000). Maps, pictograms, and other early-emerging graphical technologies all have something in common in that each abstracts away from a more complex physical or psychological reality in order to provide a “boiled down” representation that still manages to preserve something about the fundamental structure of the thing it represents. While the term schema has made several appearances in the history of philosophy, neurology, psychology, and linguistics (see Amorapanth et al., 2012 for a more thorough discussion), here the term is used to describe any iconic representation where perceptual detail has been abstracted away from reality in order to provide a more flexible representation for cognition. For example, an iconic smiley face is better than a photograph of a smiling person for representing happiness because it can be mapped more flexibly across individuals and contexts (McCloud, 1993). As such, a schema occupies an intermediate position on a continuum between very concrete perceptual complexity and very abstract symbolic representation (Chatterjee, 2001) (Fig. 1). Unlike truly symbolic representations of spatial relations (i.e., prepositions) a schema preserves some of the analog spatial qualities of the thing it stands for (Deacon, 1997; Peirce, 2011). In this sense, schemas function more like percepts. However, like words, schemas may be more likely to be processed holistically and function in a quasi-symbolic way. A recent patient group study using voxel lesion symptom mapping techniques (VLSM) identified left hemisphere regions associated with verbal representations and discrete right Fig. 1 e Schemas as an intermediate representational format. Schemas preserve the analog qualities of a spatial relation but may also function like symbols. In a schema, the actual spatial relation is inherent to the representation used to communicate spatial meaning. However, unlike perceptually rich analog representations, schemas put less of a load on the attentional system. As such, quasisymbolic schemas, like words, may be processed as single objects. The intermediate “best-of-both-worlds” representational status of schemas suggests that they may be of particular use to a simultanagnosic with a compromised attentional system and impaired spatial processing abilities. hemisphere structures for representing pictures and schemas using a spatial relations matching task similar to the one constructed for the present study (Amorapanth et al., 2012). The parietal lobe is known to play a critical role in representing spatial relations among objects as part of the dorsal visual pathway (Ungerleider and Haxby, 1994). Patients with lesions to parietal cortices typically show profoundly impaired spatial abilities. Specifically, bilateral parietal lesions often result in simultanagnosia, a condition in which patients are characteristically unable to perceive more than a single object at a time (Luria, 1959). Studying individuals with simultanagnosia and related syndromes can further our understanding of normal visual processing and its underlying neural basis (Coslett and Chatterjee, 2003; Coslett and Lie, 2008). Simultanagnosia is often regarded as an attentional disorder; a kind of bilateral neglect. Presently, simultanagnosia presents an interesting case for the investigation of schemas. If schemas help us to abstract spatial relations from complex scenes, and aid relational thinking, they might be especially helpful for an individual with simultanagnosia. Schemas are designed to make basic spatial relations more salient and are visually less complex than perceptually rich scenes. The simplified visual input inherent to schemas may put less of an attentional demand on a compromised system as compared to more sensorially loaded stimuli. Some recent evidence suggests that increasing the saliency of spatial relations can improve performance for simultanagnosics on tasks that interrogate global processing abilities (Huberle and Karnath, 2010) as can priming simultanagnosic patients with low-spatial frequency stimuli (Thomas et al., 2012) or taskrelevant semantic information (Soto and Humphreys, 2009). The present study investigates differences between particular formats in comprehending spatial relations. 2. Methods 2.1. Participants Patient E.E. (female, 43 years old, 18 years education) experienced three parietal lobe infarcts between May and June of 2004. These events resulted in bilateral lesions extending from the occipital lobes to middle parts of the inferior parietal sulcus (See Fig. 2A). Lesions did not extend into precuneus or superior parietal lobe. Behavioral testing indicated simultanagnosia. E.E. was unable to comprehend more than one object simultaneously 30 months after her most recent stroke. For example, when presented with the two images in Fig. 2B she reported seeing either a “red daisy” (2B left) or a green oval (2B right), unable to perceive the yellow and blue ovals. E.E. also showed a 100% local bias with Navon Letters. E.E. has normal visual acuity, color and shape perception, and object recognition (Berryhill et al., 2009). E.E.’s simultanagnosia greatly affects her everyday living: E.E. self reports problems with her spatial awareness in everyday life. For example, she states that she frequently bumps into furniture and doorways. She also describes events such as reaching for something and either knocking it over or having to continue to reach for it because she c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 1985 Fig. 2 e A. MRI images for patient E.E. showing lesion sites. Axial slices were obtained from clinical scans. Bilateral lesions (as indicated by areas of hypodensity) ranged from the occipital lobes to middle parts of inferior parietal sulcus without extending into precuneus or superior parietal lobes. B. Occlusion stimuli adapted from Berryhill et al. (2009). When presented with the two images, E.E. reported seeing either a “red daisy” (left) or a green oval (right), unable to perceive the yellow and blue ovals. misjudged proximity. In other words, she experiences spatial difficulties even in well-known environments such as her own home. One strategy she uses for overcoming this problem is to keep objects in known locations and to stay in familiar places (Berryhill, 2012). (See also, Berryhill et al., 2009, 2011) An age and education matched control group of neurologically healthy participants were also tested on identical tasks (N ¼ 5; meanage ¼ 51.4 years, meaneducation ¼ 17 years). All control participants were native English speakers, right handed, with normal or corrected to normal vision. Written informed consent in accordance with the procedures of the Institutional Review Board of the University of Pennsylvania was obtained for all participants in this study. 2.2. Stimuli The stimuli and tasks were similar to those used in a previous group study (Amorapanth et al., 2012). The stimuli were designed to assess understanding of basic spatial meanings. Typically, such meanings are communicated verbally by locative prepositions. Locative prepositions describe figuree ground relations between at least two objects: the reference object (ground) and located object (figure). Four prepositions served as the words in each matching task and were selected to include two main locative preposition classes as described in the literature (Talmy, 2000). Topologic prepositions describe figureeground relations that vary along the dimensions of contact and degree of enclosure (i.e., object-based spatial relations like IN and ON); and projective prepositions describe figureeground relations that vary along the dimensions of vertical or horizontal displacement (i.e., axis-based spatial relations like ABOVE and BELOW). Each matching task used these 4 spatial concepts. Practically, topological relations involve objects that to some degree occlude one another whereas projective relations involve objects that are distally separated in space. In principle, both kinds of relations should be difficult for a patient with simultanagnosia. In all image stimuli (pictures and schemas), the figure object was red and the ground object was non-red. For the pictures in each matching task, realistic color photographs were used as stimuli. Pictures were designed to unambiguously depict the same spatial relations as denoted by the prepositions. The objects in these pictures consisted of a small set of relatively common household or office items that could function as the figure or ground object for the locative relations being tested (e.g., a pair of scissors, a mug, a fork, a cutting board). As much as possible, we used the same objects, arranged in different ways, to depict distinct lexicalized spatial relations. We constructed schemas consisting of simple lines and geometric forms using graphic-making tools in Photoshop. Each set of four schemas varied along parameters proposed by Talmy (2000), such as containment, support, and degree of separation. After the construction of 96 unique schemas and 155 pictures, fifteen normal adults participated in a norming study to select the least ambiguous images for depicting each of the four spatial relations. Rating tasks (i.e., “How well does the red object’s relationship to the black object illustrate the meaning of the preposition”) and forced choice tasks (i.e., “Select the schema/picture where the red object’s relationship to the black object illustrates the meaning of the given preposition”) were used. The fifteen best schemas and pictures for each spatial relation (IN, ON, ABOVE, & BELOW) were selected for a total of 60 unique schemas and 60 unique pictures. 2.3. Procedure 2.3.1. Spatial matching tasks Incorporating the three basic types of stimuli described above (words, pictures, and schemas) six matching tasks were used: 1986 c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 (1) word to picture; (2) schema to picture; (3) word to schema; (4) picture to picture; (5) schema to schema; and (6) picture to schema (See Fig. 3). These six tasks comprise all interesting combinations of matching to and from each image format. (Matching to words was deemed uninteresting, as there could be limited variation in items between trials.) All tasks required participants to match a relation depicted in a probe item to one of four target items. Among the four targets to choose from, one correctly depicted the spatial relationship in the probe image, one depicted a within-class relation, and two depicted across-class relations (projective versus topological). Foils were distributed as such in all four tasks. Practically, this meant that the target items always depicted an IN, ON, ABOVE and BELOW relation. For each matching task, subjects indicated which one of four pictures or schema targets depicted the same spatial relation as the probe either by pointing (E.E.) or by reading a letter underneath a particular image (controls). Each task began with 5 practice trials followed by 80 real trials. Individual probe items depicted one of four discrete spatial relations (IN, ON, ABOVE, & BELOW) and were balanced within participants. Probe items could be either one of four words, or one of 60 unique schemas or pictures. Particular schemas and pictures were randomly assigned to each probe and target location for a given trial without repetition within a trial. Between trials, schemas and pictures could be repeated in either probe or target locations. All tasks in the present study used two spatial relational concepts representing topological relations (IN or ON) and two representing projective relations (ABOVE or BELOW). Matching tasks were blocked and presented using an interrupted time series design. In a variation of an AeBeAeB design typically used for single case studies, the current study used an AeBBeAeBB design. This allowed us to examine three interesting transitions (AeB1; BeA; AeB2) between baseline tasks (A) with no schemas (Fig. 3A) and intervention tasks (B) with schemas (Fig. 3B) while also providing the means to assess any possible cumulative effects of schemas on performance (BB1; BB2). Tasks were self-paced and without time constraints. 3. Results Accuracy results are summarized in Fig. 4. Overall (Fig. 4A), controls outperformed E.E. on all tasks [control group task means: 81%, 96%, 96%, 99%, 98%, 99%, respectively; t(79), p’s < .01 (Crawford and Garthwaite, 2007)]. On the two tasks that did not include matching to, or from, schemas [word-topicture (wordepic); picture-to-picture (picepic)], E.E. performed at chance levels [20% and 25%, respectively; c2 (1, N ¼ 80), all p’s > .3]. For the tasks with schemas [schema-topicture (schemaepic); word-to-schema (wordeschema); schema-to-schema (schemaeschema); picture-to-schema (piceschema)] performance was significantly better than chance [50%, 74%, 67%, 84% respectively, c2 (1, N ¼ 80) all p’s < .01]. Because tasks were administered in the sequence depicted in the graphs, Fig. 4 also provides insight regarding possible learning effects over the course of testing. As E.E. performed at chance for the first task (wordepic) and then appeared to improve gradually on the following two tasks (schemaepic and wordeschema), it is possible that a learning effect could explain the observed differences in accuracy. However, the subsequent “recovery” of impairment in the only other matching task that did not include schemas (picepic) argues against this possibility. E.E. is at chance performance only on the two baseline tasks (picepic and wordepic) suggesting that the inclusion of schemas on matching tasks made spatial relational matching possible. Better performance for Fig. 3 e Example trials for each of the six types of matching tasks used in the study. An interrupted time series design was used (AeBBeAeBB). There were two matching tasks which provided baseline scores on spatial matching without schemas (A). There were four kinds of intervention tasks with schemas (B). Numbers indicate the sequential order of presentation in the actual experiment for both E.E. and controls. c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 1987 Fig. 4 e Results. A. Overall accuracy across all tasks for E.E. and control group (N [ 5). Error bars indicate standard deviation. E.E.’s accuracy was at chance levels for the two tasks that did not include schemas. B. E.E.’s performance on topological versus projective trials. E.E. on the schemaepicture versus wordepicture task suggests that schemas make spatial relations visible for E.E. in a manner that words cannot. E.E.’s spatial matching performance increases with each second exposure to schemas [wordeschema > schemaepic, t(79) ¼ 4.02, p < .01; piceschema > schemaeschema, t(79) ¼ 3.07, p < .01] generating the hypothesis that schemas may have a cumulative facilitatory effect (see Discussion below). Fig. 4B depicts a comparison of E.E.’s accuracy on topological relations (IN and ON trials) versus projective relations (ABOVE and BELOW trials). Inspecting these data, there is no significant difference between levels of accuracy for topological versus projective relations for the two tasks that did not include schemas (wordepic and picepic). This further argues against a general learning effect: performance appears to be qualitatively different between wordepic and schemaepic tasks (tasks 1 and 2) and similar between wordepic and picepic tasks (tasks 1 and 4). For the second and third tasks presented, fewer errors occur on topological as compared to projective trials [schemaepic and wordeschema; t(1), p’s < .01]. This topological advantage for the first two tasks with schemas is not present for the last two tasks with schemas. We speculate that schemas might initially facilitate matching for topological relations more than for projective ones. The hypothesis that schemas have a differential effect on the learning of topological and projective relations would require further experimentation to be tested adequately. 4. Discussion Schemas appear to make spatial relations visible for a patient with simultanagnosia. Patient E.E. performed at above chance for all spatial relational tasks with schemas and performed at chance for all tasks without them. These data also suggest that E.E. initially had more difficulty with projective compared to topological spatial relations. Projective relation trials required her to locate two objects distally separated from one another in space. Topological relation trials required her to locate the relative positions of two objects despite some degree of occlusion. It is not simply the case that E.E. could see spatial relations in schematic form where she could not see spatial relations in photographs. Seeing spatial relations in schemas actually made it possible for E.E. to see spatial relations between objects in more perceptually rich picture stimuli. This means that the recovery of global processing observed in E.E. was not simply the result of manipulating particular features of stimuli like shape and proximity to make object relations more salient (Huberle and Karnath, 2006). Furthermore, verbal descriptions of spatial relations alone did not have this effect. In the present study, access to spatial semantics was not sufficient to prime spatial attention (Soto and Humphreys, 2009). It seems that because schemas are “abstract but analog” they are particularly good at representing spatial relations in a meaningful and perceptually salient manner. Not only was E.E. able to see spatial relations when such relations were presented in a schematic format, schemas aided E.E. in seeing spatial relations when they were presented in more perceptually rich image formats. A relatively low-level visual process may provide the underlying mechanism for this effect. A recent study (Thomas et al., 2012) found that making low-spatial frequency information available to simultanagnosic patients enabled global processing. Spatial deficits associated with simultanagnosia could result from impaired processing of low-spatial frequencies in damaged areas of the dorsal visual pathway. By 1988 c o r t e x 4 9 ( 2 0 1 3 ) 1 9 8 3 e1 9 8 8 “biasing” low-spatial frequency processing in the intact ventral pathway, visuospatial attention and perception can be recovered. Thomas et al. (2012) used psychophysical techniques to engage parvocellular networks for low-spatial frequency processing (using parvocellular-biased stimuli with high-color contrasts). In the present study schemas may have biased neural processing for E.E. in such a manner, allowing her to attend more globally to spatial relations within visually complex stimuli by making low-spatial frequency information more salient to areas undamaged by stroke. The schemas used in the present study used high contrast colors to denote meaningful figureeground relations. In support of this explanation, across the six matching tasks exposure to schemas appears to have a cumulative facilitatory effect. E.E.’s spatial matching performance increases with each second exposure to schemas (wordeschema > schemaepic; piceschema > schemaeschema) even after being “reset” to baseline chance performance over the picture-to-picture matching condition. While more research is needed before drawing strong conclusions, the Thompson et al. study (2012) and the present results generate the hypothesis that targeted training could reduce symptoms in simultanagnosic patients. If schemas make spatial relations more visible to a patient with simultanagnosia, simplified graphical representations like graphs and diagrams may serve a similar function for neurologically intact individuals. Graphical devices are designed to communicate the gist of a more complex reality and as such are often what allow us to “see the forest before the trees.” Good graphs and diagrams may naturally engage lowspatial frequency processes that encourage global visual processing allowing individuals to perceive parts as they relate to the whole. The results from this case study provide general insight regarding how schemas facilitate spatial reasoning when used in graphic depictions, and how such theoretically intermediate representational structures relate to perceptual and verbal representations of categorical spatial relations. It is our position that schemas, like those used in graphical technologies, can be construed as intermediate representational structures. Mental representations that code spatial relations at an analogous level of abstraction and informational complexity may play some role in linking percepts and verbal representations in the brain. A prior study (Amorapanth et al., 2012) found evidence that when representing spatial relations during similar matching tasks to those used here, the brain distinguishes between rich percepts, schemas, and words. This makes sense considering that schematic graphical technologies have been used for communicating complex ideas by most cultures starting before the invention of writing. Acknowledgments The authors thank Bianca Bromberger, Jonathan Yu, Eileen Cardillo, and Laurent Turgeon-Dharmoo for help in creating stimuli, preparing figures, and copy editing. The authors are particularly grateful to E.E. for participation, and Marian Berryhill for discussion. This research was supported by a subcontract to AC from the National Science Foundation: Spatial Intelligence and Learning Center: SBE-0541957. references Amorapanth P, Kranjec A, Bromberger B, Lehet M, Widick P, Woods A, et al. Language, perception, and the schematic representation of spatial relations. Brain and Language, 120(3): 226e236, 2012. Berryhill ME, Chein J, and Olson IR. At the intersection of attention and memory: The mechanistic role of the posterior parietal lobe in working memory. Neuropsychologia, 49(5): 1306e1315, 2011. Berryhill ME, Fendrich R, and Olson IR. Impaired distance perception and size constancy following bilateral occipitoparietal damage. Experimental Brain Research, 194(3): 381e393, 2009. Berryhill ME. Personal Communication, 2012. Chatterjee A. Language and space: Some interactions. Trends in Cognitive Science, 5: 55e61, 2001. Chippindale C and Nash G. The Figured Landscapes of Rock-art: Looking at Pictures in Place. Cambridge, UK: Cambridge University Press, 2004. Coslett HB and Chatterjee A. Balint’s syndrome and related disorders. In Feinberg TE and Farah MJ (Eds), Behavioral Neuroscience and Neuropsychology. New York: McGraw-Hill, 2003: 325e335. Coslett HB and Lie G. Simultanagnosia: When a rose is not red. Journal of Cognitive Neuroscience, 20(1): 36e48, 2008. Crawford JR and Garthwaite PH. Comparison of a single case to a control or normative sample in neuropsychology: Development of a Bayesian approach. Cognitive Neuropsychology, 24(4): 343e372, 2007. Deacon TW. The Symbolic Species: The Co-evolution of Language and the Brain. WW Norton & Company, 1997. Gaur A. A History of Writing. British Library, 1985. Huberle E and Karnath HO. Global shape recognition is modulated by the spatial distance of local elements e Evidence from simultanagnosia. Neuropsychologia, 44(6): 905e911, 2006. Huberle E and Karnath HO. Saliency modulates global perception in simultanagnosia. Experimental Brain Research, 204(4): 595e603, 2010. Kranjec A and Chatterjee A. Are temporal concepts embodied? A challenge for cognitive neuroscience. Frontiers in Psychology, 1(240): http://dx.doi.org/10.3389/fpsyg.2010.00240, 2010. Luria A. Disorders of simultaneous perception in a case of bilateral occipito-parietal brain injury. Brain, 82(3): 437e449, 1959. McCloud S. Understanding Comics: The Invisible Art. Harper Paperbacks, 1993. Peirce CS. Philosophical Writings of Peirce. Dover Publications, 2011. Smith C. The emergence of ‘maps’ in European rock art: A prehistoric preoccupation with place. Imago Mundi, 34: 9e25, 1982. Soto D and Humphreys GW. Semantically induced distortions of visual awareness in a patient with Balint’s syndrome. Cognition, 110(2): 237e241, 2009. Talmy L. Towards a Cognitive Semantics: Concept Structuring Systems. MIT Press, 2000. Thomas C, Kveraga K, Huberle E, Karnath HO, and Bar M. Enabling global processing in simultanagnosia by psychophysical biasing of visual pathways. Brain, 135(5): 1578e1585, 2012. Tversky B, Zacks J, Lee P, and Heiser J. Lines, blobs, crosses, and arrows: Diagrammatic communication with schematic figures. In Anderson MM, Cheng P, and Haarslev V (Eds), Theory and Application of Diagrams. Berlin: Springer-Verlag, 2000: 221e230. Ungerleider LG and Haxby JV. ‘What’ and ‘where’ in the human brain. Current Opinion in Neurobiology, 4(2): 157e165, 1994.