Exp Brain Res (2007) 181:199–211 DOI 10.1007/s00221-007-0923-2 R ES EA R C H A R TI CLE Scan patterns during the processing of facial identity in prosopagnosia Jason J. S. Barton · Nathan RadcliVe · Mariya V. Cherkasova · Jay A. Edelman Received: 4 July 2006 / Accepted: 19 February 2007 / Published online: 15 March 2007 © Springer-Verlag 2007 Abstract The scan patterns of ocular Wxations made by prosopagnosic patients while they attempt to identify faces may provide insights into how they process the information in faces. Contrasts between their scanning of upright versus inverted faces may index the presence of a hypothesized orientation-dependent expert mechanism for processing faces, while contrasts between their scanning of familiar versus novel faces may index the inXuence of residual facial memories on their search for meaningful facial information. We recorded the eye movements of two prosopagnosics while they viewed faces. One patient, with acquired prosopagnosia from a right occipitotemporal lesion, showed degraded orientation eVects but still with a normal distribution of Wxations to more salient facial features. J. J. S. Barton · M. V. Cherkasova Department of Neurology, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA However, the dynamics of his global scan patterns were more chaotic for novel faces, suggesting degradation of an internal facial schema, and consistent with other evidence of impaired face conWguration perception in this patient. His global scan patterns for famous faces diVered from novel faces, suggesting the inXuence of residual facial memories, as indexed previously by his relatively good imagery for famous faces. The other patient, with a developmental prosopagnosia, showed anomalous orientation eVects, abnormal distribution of Wxations to less salient regions, and chaotic global scan patterns, in keeping with a more severe loss of face-expert mechanisms. The eVects of fame on her scanning were weaker than those in the Wrst subject and non-existent in her global scan patterns. We conclude that scan patterns in prosopagnosia can both reXect the loss of orientation-dependent expert mechanisms and index the covert inXuence of residual facial memories. In these two subjects the scanning data were consistent with other results from tests of conWguration perception, imagery, and covert recognition. J. J. S. Barton Division of Neurology, Department of Ophthalmology and Visual Sciences, University of British Columbia, Vancouver, BC, Canada Keywords Face recognition · Inversion · Scanpath · Morph · Eye movement · Prosopagnosia J. A. Edelman Department of Psychology, Harvard University, Cambridge, MA, USA Introduction N. RadcliVe Faculty of Medicine, Temple University, Philadelphia, PA, USA J. J. S. Barton (&) Neuro-ophthalmology Section D, VGH Eye Care Center, 2550 Willow Street, V5Z 3N9 Vancouver, BC, Canada e-mail: jasonbarton@shaw.ca Inspection of objects or scenes is usually accomplished with a series of scanning saccades that shift ocular Wxation from one location to another. It is unlikely that these Wxations are randomly distributed across the image: rather, their purpose may be to direct both the locus of attention and the high spatial resolution of the macula to regions that will yield information of use to the observer. From this goal-directed stance, scanning acquires data for a cognitive 123 200 process aimed at reaching perceptual decisions about the world (Rybak et al. 1998; Deco and Schurmann 2000; Henderson 2003). Studying scan patterns can tell us where useful information is located in a scene. Furthermore, because scanning is a quintessentially serial process, scan patterns can also inform us about the temporal organization of information processing within the observer. This concept of scanning as both a manifestation of and an instrument for information processing is consistent with observations that scan patterns reXect properties of both the stimulus and the observer. Attention and Wxations can be attracted by properties such as color contrast, high-contrast edges, and relative motion, which may indicate regions in the scene containing high-yield information. On the other hand, the goals of the observer may alter scanning: the same observer will scan a city scene diVerently when looking for an address than when looking for a person in the crowd. The observer’s prior experience and expertise with stimuli can also inXuence scan patterns. Art experts scan pictures diVerently from naïve observers (Zangemeister et al. 1995), for example, and experimental subjects scan familiar items diVerently than novel items (AlthoV and Cohen 1999). These illustrate important ‘top-down’ eVects of the cognitive status of the individual upon the scan patterns used to acquire information from a scene. Because of these observer-dependent eVects, it may be possible to use scan patterns as a probe of the perceptual capabilities of individuals with or without neurological disorders. Patients with prosopagnosia, for example, have a speciWc dysfunction that impairs their ability to recognize familiar faces. In some of these individuals the deWcit may lie at a perceptual level, degrading their ability to acquire perceptual data about faces (Barton et al. 2002). This perceptual function is a highly expert process, able to discern the very subtle changes that distinguish one speciWc face from the thousands of other faces encountered during a lifetime. If this expert mechanism is lost, it may be possible to discern its absence in altered scan patterns of prosopagnosic patients viewing faces. Other literature from normal subjects suggests that one of the hallmarks of a face-expert processing mechanism is the inversion eVect. Faces are more diYcult to recognize upside down (Yin 1970), likely because inversion impairs the perception of facial structure (Sergent 1984; Kemp et al. 1990; Searcy and Bartlett 1996; Leder and Bruce 1998; Barton et al. 2001b, 2003b; Malcolm et al. 2004). The dual-mode hypothesis (Bartlett and Searcy 1993; Farah et al. 1998) explains this by postulating that faces can be processed by both generic object recognition and a trained expert mechanism; however, since expertise is normally acquired through repeated exposure to upright faces, the trained expert mechanism develops an orientation dependency, so that it is operative with only upright but not 123 Exp Brain Res (2007) 181:199–211 inverted faces. Our prior work showed that normal subjects scan upright faces diVerently than inverted faces, with less overall scanning, less scanning of the less salient lower face, and a less random sequence of Wxations (Barton et al. 2006). If an orientation-dependent expert mechanism for looking at faces is disrupted in prosopagnosia, then their scan patterns may fail to show these diVerences between upright and inverted viewing of faces. In addition to showing eVects of expertise, scan patterns may also show eVects of experience, or familiarity. Wellknown faces possess representations within the subject’s store of facial memories, and this knowledge may guide the exploratory scan patterns of subjects as they engage in the process of identifying the face. Contrasts between famous and recently encountered (novel) faces have shown that familiarity generates a variety of eVects on scan patterns (AlthoV and Cohen 1999; Stacey et al. 2005; Barton et al. 2006). In prosopagnosia, such scanning diVerences between famous and novel faces could index the residual eVects of surviving stores of facial memories, even if the patients were not able to recognize the faces being shown. Such a dissociation between absent overt recognition and residual behavioral eVects that distinguish between familiar and novel faces could also be considered a form of covert recognition in this disorder (Bruyer 1991; Young 1994). Indeed there is one previous report of such diVerences in the face scan patterns in two prosopagnosic patients (Rizzo et al. 1987). In this study we had subjects engage in a face identiWcation task while we manipulated the variables of expertise (upright versus inverted) and familiarity (famous versus novel). We used a series of morphed faces to increase the level of diYculty in the identiWcation task for control subjects, and hence encourage the generation of scan patterns during a more protracted period of information processing. We studied two prosopagnosic patients, one with an acquired lesion and one with the developmental form. Our goal was to determine if either subject demonstrated residual expertise in the perceptual processing of faces, or residual familiarity eVects suggesting the presence of facial memories and a form of covert recognition. Methods Subjects Two prosopagnosic patients participated in the study. Subject 002(D) is a 31-year-old woman with generalized seizures since 2 years of age. She has always had diYculty recognizing faces, using cues such as voice or clothing instead. She also has trouble recognizing her brother’s car in parking lots. She gets lost in familiar surroundings and Exp Brain Res (2007) 181:199–211 her reading is slow, but she denied problems with color perception. Her visual acuity was 20/50. She saw 11 of 14 Ishihara pseudoisochromatic plates, in the normal range. Confrontation visual Welds were only mildly constricted in the periphery. She had left superior oblique overaction but otherwise normal eye movements. Neurologic examination was normal. Neuropsychologic testing showed average verbal intelligence and superior attention and working memory. Memory for verbal material was normal, but that for non-verbal material was severely impaired. Object recognition was impaired, for unusual views and line drawings. Reading speed and comprehension were slow. There was no evidence of hemi-neglect. On a test of discrimination between familiar and unfamiliar faces, she scored a d’ of zero, indicating no discriminative ability. Forced-choice tests also showed no evidence of covert recognition (Barton et al. 2001a). Her perception of facial conWguration was severely impaired (Barton et al. 2003a). MRI showed abnormal gyral patterns of both occipital lobes, interpreted as a mild focal developmental polymicrogyria (Fig. 1, top). Subject 005(A) is a 52-year-old man seen 7 months after a right occipital hemorrhage from an underlying oligodendroglioma. This was resected and he was treated with irradiation and chemotherapy. He complained of blurry vision and diYculty reading. He had trouble recognizing buildings in his hometown, and got lost occasionally. Snellen acuity was 20/25 in either eye, and he had a complete left homonymous hemianopia on Goldmann perimetry. He saw 12/14 Ishihara pseudoisochromatic plates. Pursuit and saccadic eye movements were normal. There were no signs of left hemi-neglect 201 or simultanagnosia. On the test of famous face familiarity, he scored a d’ of 1.12, indicating very weak overt recognition of faces (Barton et al. 2001a). His perception of facial conWguration was severely impaired (Barton et al. 2002), but his imagery for faces was normal, particularly for facial features, suggesting that facial memory stores were intact (Barton and Cherkasova 2003). He had evidence of covert recognition on both forced-choice tests (Barton et al. 2001a) and an indirect priming method (Barton et al. 2004). MRI showed an extensive right medial occipital resection involving the lingual and fusiform gyri (Fig. 1, bottom). As controls, we studied eight normal subjects, three female and Wve male, with a mean age of 31.5 years (range 23–41 years). All subjects gave informed consent and the institutional review board of Beth Israel Deaconess Medical Center approved the protocols. Images We created a set of eight morph series, where each series represented a gradual physical transformation from one face to a second. Four series were transitions between two male faces, and four were transitions between two female faces. In each gender set, two series used pairs of famous faces and two used pairs of novel faces. For famous faces, we selected eight images with a highresolution frontal view from the internet. We rejected images with excess jewelry, make-up, facial hair, open mouths, or glasses. All subjects, both prosopagnosic and control, acknowledged having heard of and having had the Fig. 1 MR scans of patients. Top Wgure is an axial T2-weighted image from subject 002(D), showing the polymicrogyria in the medial occipital cortex (arrows). Bottom Wgures are coronal T1-weighted images from subject 005(A), with gadolinium enhancement in the right image. These show the medial occipitotemporal aspects of the lesion, involving the general vicinity of the fusiform face area (arrows) 123 202 opportunity to see the faces of these celebrities. In particular, the two patients were able to provide semantic information about the celebrities in response to their name, even though they were not able to identify their faces or recognize the faces as being familiar. For novel faces, we used a digital camera to take pictures of eight anonymous individuals, ensuring that their images matched the above criteria for famous faces. Using Photoshop 5.5 (Adobe, San Jose, CA, USA), we replaced the backgrounds with a black surround, removed earrings, and cropped the images to eliminate the external hair contour and neck. Morph pairs were matched for hair color, facial shape, image size, image luminance, sex, and fame. The celebrity pairs were Demi Moore/Julia Roberts, Michelle PfeiVer/Gwyneth Paltrow, John Travolta/Paul Newman, Matthew Broderick/Kevin Costner. For each morph series the two base (unmorphed) images were loaded into Morph Studio 1.0, where 120 points were matched between each image on analogous facial regions (i.e., center of pupils, corner of mouth). We created a morph series of 21 images, representing a 5% change in position, hue and contrast between the two base images (designated images 1 and 21). From each series we used the two base images (images 1 and 21, representing mixtures of {0:100}) and Wve morphed images from near the center of each series (images 9, 10, 11, 12, and 13, representing mixtures of {40:60}, {45:55}, and {50:50}). Each image was set to a height of 330 pixels, and because face shapes varied, widths ranged from 299 pixels to 392 pixels. These generated relatively large images, which improve the accuracy of locating Wxations to a given region of interest. Cropping of hair was performed to enforce attention to facial details. It is possible that the use of large images, omission of hair contour, and the use of morphed images limit the ability to compare these data to other studies, but we believe these maneuvers enhanced the accuracy of the results and focus the eye movements upon the face processing task at hand. Eye movement protocol We recorded eye movements with a magnetic search coil technique, using a scleral contact lens and a three-foot Weld coil (Crist Instruments, Bethesda, MD, USA). The subjects’ heads were secured in a chin rest. The lens was placed in the left eye. We calibrated the system by having subjects Wxate nine targets in a square grid spanning 50°. Twelve data points were collected at each of nine targets, and a regression method was used to Wnd the best linear Wt. Eye position was digitized at 500 samples/s. Displays were generated by a Power Macintosh 9600/ 233 using programs written in C++ on the Vision Shell programming platform (www.kagi.com/visionshell), and 123 Exp Brain Res (2007) 181:199–211 back-projected onto the tangent screen with an Eiki LC7000U projector. Distance from the subject’s cornea to the screen was 80.5 cm. On the screen, each facial image spanned a height of 67° with widths ranging from 61° to 82°. We divided our faces into two experimental sets, one male and one female. Before recordings, we showed the subjects four panels, each containing an array of eight faces, all female or all male. Two of these arrays showed the base images used for the morphed series, while the other two showed alternate pictures of the same individuals. The faces had the names of the individuals printed below them, and subjects were asked to try to remember the identity of the faces. They were allowed to study these panels as long as they wished. They were told that in the experiment they would be seeing a sequence of faces that had been altered, and that they would be asked to identify to whom each face belonged. We presented our faces in two blocks, one female and one male. The order of male/female blocks was counterbalanced across the eight subjects. Within each block, half the trials were upright and half inverted. In each block, morph series (e.g., Demi Moore/Julia Roberts), morph image (e.g., 40Demi:60Julia), and orientation were randomly ordered. Each face image was seen just twice, once in each orientation. Thus each subject saw a total of 112 faces (seven morph levels, four same-sex face pairs, two orientations, two diVerent-sex blocks). A trial began with a Wxation screen with a small red spot at center. Fixation within 3° of this zone triggered presentation of a stimulus face, along with a red spot 30° below screen center. Subjects were asked to look at this second red spot when they had reached a decision, then state whether this face was famous or novel. On their report the collection of eye movement data stopped and the single face was replaced with an array of the eight alternate upright facial images–(i.e., the faces not used as base images in the morphed series. These alternates were used to emphasize the identity recognition element of the task, rather than pattern matching). Each of the eight faces had a number below it and the subject indicated the number of the individual whose face they thought they had just seen. When they had done so, the panel vanished, the Wxation screen for the next trial appeared, and collection of eye movement data resumed. After every ten trials a calibration screen appeared, with Wve Wxation spots, one at center and one at the four corners of the region where the faces were displayed. These were numbered, and subjects were asked to look at each spot in order, and then to return to screen center. Data analysis Using Matlab (Mathworks, Natick, MA, USA), we identiWed saccades as eye movements with velocities exceeding Exp Brain Res (2007) 181:199–211 47°/s. The onset of a saccade was the point at which the velocity of the eye Wrst exceeded 31°/s, and the end of a saccade as a point where velocity fell below this baseline. Fixations were deWned as the intervals between the end of one saccade and the beginning of the next saccade. We characterized each Wxation by its horizontal position, vertical position, duration, and its time of onset relative to the start of the trial. We recorded all Wxations from the beginning to end of each trial. The initial Wxation, evident as a long Wxation of several seconds at screen center, was discarded, and the Wrst Wxation after this long Wxation was taken as the onset of the scanning sequence. The end of the scanning sequence was the point at which subjects Wxated the red dot at 30° below screen center. Not all trials were terminated by such a Wxation, however, as subjects often forgot to Wxate this peripheral spot when they had made a familiarity decision. In these trials, a single long Wxation of more than 1,500 ms duration was often present near the end of the string of eye movement data, usually at an eccentric position in relation to the image, and this was taken to indicate the end of the subject’s scanning. In the absence of either a Wxation at the red spot or a long nearterminal Wxation, we included all Wxations occurring to the end of data collection for a trial. Our next step was to adjust our data for any baseline drift or gain change that might have occurred during recording, since absolute eye position was critical to our results. From the recurring calibration trials we obtained the average eye position for Wxations near each of the Wve spots. OVset shift was calculated as the mean diVerence in x and y positions from the expected value for each point. Gain shift was calculated as the mean diVerence in distance of each of the four corner Wxations from the Wxations at screen center (gain shift proved to be negligible for all data sets). We derived the oVset and gain correction required by averaging these calculations from the two calibration trials sandwiching a set of ten face trials, and applied to all ten trials. We then collated the corrected data from all subjects for each trial, displayed these data on an x–y plot of position upon the facial image of that trial, and identiWed all Wxations that fell upon the face. Our analysis of scanning could be divided into three sections: a. Total scanning–how much scanning did the subjects devote to the diVerent faces in general? For each subject we calculated the total number of Wxations and total scan time (the sum of the durations of all Wxations) used to scan each face. We also analyzed the mean duration of individual Wxations to determine if any change in total duration was due to an increase in number of Wxations or the length of Wxations. 203 b. Regional distribution of scanning—where did they scan within the faces? We divided each face into eight diVerent feature regions: brow, left eye, right eye, nose, mouth, chin, right cheek, left cheek (Fig. 2) (Groner et al. 1984). Approximate criteria were used uniformly to draw the boundaries between regions for diVerent faces. Fixations were classiWed according to their region. As a secondary comparison we contrasted upper (brow and eyes) with lower face (nose, mouth, chin, and cheeks), and right hemi-face (right eye and cheek) with left hemi-face (left eye and cheek). As with total scanning, we analyzed both number of Wxations and scan time per region. c. Scanning dynamics—how did they scan faces? To analyze the sequence of Wxations between diVerent facial regions, we performed two separate analyses. Both were directed at determining the variability of scanning sequences. The Wrst analysis examined the local predictability of scanning using Markov Wrst-order matrices (Rizzo et al. 1987; Gbadamosi and Zangemeister 2001). We created a variable called ‘pairs’, where the Wrst digit represented the facial region where the prior Wxation had been located, and the second digit the facial region of the current Wxation. With eight diVerent regions, this creates 64 possible pairs. These can be formatted as an 8 £ 8 Wrst-order Markov matrix, in which the rows represent the Wrst Wxation and the columns represent the second Wxation. The asymmetric lambda (aL) is a measure of the predictability or non-random nature of the pairs within this matrix. One particular sequence pair of interest is the ‘locally repetitive pair’. These are pairs where both Wxations are in the same region. Whether these locally repetitive Wxation pairs should be included in analyses of scan sequencing is contentious. Others have used ‘compressed’ Markov matrices that omit locally repetitive pairs (Gbadamosi and Zangemeister 2001). We too used compressed Markov matrices, which better assess the inter-regional Xow of information processing. This yielded compressed asymmetric lambda values (aLc). We calculated eight aLc values for each subject, grouping data with common orientation (upright versus inverted) and fame (novel versus famous). Our prior report on normal subjects found no signiWcant eVect of morphing on aLc values (Barton et al. 2006), so we eliminated this as a main eVect from the analysis. We also performed a separate analysis of locally repetitive pairs. These may represent information gathering on a diVerent scale, possibly directed at local component analysis rather than perception of the whole structure. We computed the frequency of such pairs, in the 123 204 Exp Brain Res (2007) 181:199–211 Fig. 2 Top shows an example of facial images used, in this case a morph series between Demi Moore and Julia Roberts. Bottom shows an example of facial regions. Each face is segmented into eight regions: brow, right eye (of the depicted face, not the viewer’s hemiWeld), left eye, nose, mouth, chin, right cheek, and left cheek eight diVerent conditions varying in orientation, morphing, and fame. The second analysis examined the global predictability of scanning. That is, how much did the whole scanning sequence for one face resemble that of another by the same subject? Recent scanning studies (Brandt and Stark 1997; Gbadamosi and Zangemeister 2001) have borrowed techniques from DNA sequence analysis to attempt to characterize this. The strategy is to quantify how much change is required to transform one sequence into another. One may make insertions, deletions, or substitutions, with the latter counting twice, since a substitution is the equivalent of a combined insertion and deletion. Here too we used a compressed variant that eliminated consecutive repetitions of a single area (Gbadamosi and Zangemeister 2001). The number of changes is divided by the average number of items (Wxations) in the two sequences being compared, to give the number of changes per Wxation. This is called the ‘editing cost’. With this formula, the minimum editing cost of ‘0’ indicates identical sequences, whereas the maximum editing cost of ‘2’ equals completely dissimilar sequences. d. Statistical analysis We used ANOVA with repeated measures to analyze the eVects of the diVerent factors, using JMP IN 5.1.2 (SAS Institute, www.jmpin.com). Where suYcient data was available (e.g., total scanning), this was performed within the two subjects, using orientation, fame, and morphing as the main eVects, across the eight diVerent series of faces. Otherwise, for all parameters, the data of each subject was contrasted against the controls independently, with subject group (prosopagnosic patient versus the controls) as an additional main eVect. The normal data sometimes showed negligible eVects of certain factors on certain variables (e.g., morphing and fame for aLc values in the Markov matrix analysis). Where this was present, those factors were eliminated as main eVects in the ANOVA analysis. SpeciWc planned comparisons for upright faces and 123 upright unmorphed faces were examined using linear contrasts within the ANOVA analysis, and are reported as P-values derived from t-tests. Results Total scanning As we previously reported, normal subjects use fewer Wxations and less viewing time to make identity judgments with upright faces than they do with inverted faces (Fig. 3). They also require fewer Wxations and less viewing time to make such judgments about famous faces than they do with novel faces, and this eVect for fame is similar for both upright and inverted faces. Subject 005(A) did not show any eVect of orientation. He did show a trend to a main eVect of fame on the number of Wxations (9.01 Wxations/face for famous versus 10.21 Wxations/face for novel, F(1,112) = 3.21, P < 0.077), but not on scan time. Planned contrasts between famous and novel faces showed similar trends for number of Wxations for all unmorphed faces (7.75 Wxations/face for famous versus 10.56 Wxations/face for novel, t = 1.77, P < 0.079) and for upright unmorphed faces (7.5 Wxations/ face for famous versus 11.25 Wxations/face for novel, t = 1.67, P < 0.098). Compared to the normal subjects, scan times for faces in general were signiWcantly longer for subject 005(A) (4,048 ms/face) than for the controls (1,934 ms/ face, F(1,7)=22.8, P < 0.002). Analysis of the duration of individual Wxations (rather than total scan time) showed that these were signiWcantly longer for subject 005(A) (415 ms/Wxation) than for controls (242 ms/Wxation, F(1,940) = 71.1, P < 0.0001). Also, the interaction between fame and orientation diVered between subject 005(A) and controls [F(1,940) = 6.94, P < 0.009]: subject 005(A) had longer individual Wxations with upright famous faces than with inverted famous faces or with novel faces, whereas controls showed no such relative increase for upright famous faces. Exp Brain Res (2007) 181:199–211 205 Fig. 3 Total scanning. Left graphs are for number of Wxations and right graphs show scan time, summed from the durations of all Wxations used to scan each face. Control data is shown in the top graphs, with error bars representing one standard error. Solid bars are for unmorphed faces, striped bars for morphed faces. Subject 005(A) is shown in the middle graphs and subject 002(D) in the bottom graphs. Both subjects show tendencies to less scanning for famous faces, though this was not signiWcant for subject 005(A). Subject 005(A) does not show the normal orientation eVect of less scanning with upright faces, while subject 002(D) shows an anomalous eVect of more scanning of upright faces Subject 002(D) showed a slight eVect of orientation on Wxation number [F(1,112) = 4.29, P < 0.041]—paradoxically with more Wxations on upright (8.61 Wxations/face) than inverted faces (6.77 Wxations/face)—but not on scan time. There was a signiWcant eVect of fame on scan time (1,949 ms/face for famous versus 2,743 ms/face for novel, F(1,112) = 14.2, P < 0.0003) but not on number of Wxations. A planned contrast showed signiWcantly shorter scan time for famous faces (1,874 ms/face) than novel faces (3,054 ms/face) in the upright unmorphed condition (t = 2.10, P < 0.038). The contrast with controls showed that subject 002(D) was not signiWcantly diVerent from controls in either number of Wxations or scan time per face. Subject 002(D) did show a signiWcant diVerence in the eVect of orientation on both Wxation number [F(1,940) = 11.52, P < 0.0007] and scan time [F(1,940) = 4.57, P < 0.033]: while controls scanned upright faces less than inverted faces, she showed the opposite pattern. Analysis of the duration of individual Wxations showed, as with subject 005(A), that these were signiWcantly longer for subject 002(D) (392 ms/Wxation) than for controls [242 ms/Wxation, F(1,940) = 47.19, P < 0.0001]. There was also a signiWcant diVerence in the eVect of fame [F(1,940) = 18.8, P < 0.0001]: while controls showed equivalent durations of Wxations for famous faces and novel faces (243 ms versus 235 ms, t = 0.81, P = n.s.), subject 002(D) had longer Wxation durations on novel faces than on famous faces (434 ms versus 328 ms, t = 4.3, P < 0.0001). To summarize, subject 005(A) and subject 002(D) required longer scan times to process faces, in large part not because they used more Wxations, but because their Wxations lasted longer. The presence of an orientation-dependent expert face mechanism should be revealed as orientation eVects. Subject 005(A) showed no orientation eVects and subject 002(D) showed an anomalous one, with more Wxations on upright than inverted faces. The presence of residual face recognition should be revealed as fame eVects. Subject 005(A) showed a trend to a normal fame eVect on Wxation number but not scan time, and subject 002(D) had a normal fame eVect on scan time but not Wxation number. Both also showed anomalous fame eVects on the duration of individual Wxations, with subject 005(A) showing longer Wxations on upright famous faces and subject 002(D) showing longer Wxations on novel faces. Regional distribution of scanning Our previous analysis of the control data (Barton et al. 2006) showed that the interaction of facial region and fame was modest, but the interaction between facial region and orientation was highly signiWcant (Fig. 4). Normal subjects scanned the upper face more than the lower in upright 123 206 Exp Brain Res (2007) 181:199–211 Fig. 4 Regional distribution of scanning. Conventions as in Fig. 3. Upright face data are plotted alongside inverted face data. Normal subjects scan the eyes, nose, and mouth the most, and show a bias for the eye in the left hemiWeld (right eye in upright faces and left eye in inverted faces). Subject 005(A) shows a similar distribution, but without the left hemiWeld bias. Subject 002(D) has an anomalous distribution, focusing more on the brow and cheek, as well as a right hemiWeld bias Fig. 5 Lateralized distribution of scanning. Left graphs are control subjects, with one standard error shown in error bars, middle graphs are for subject 005(A), and right graphs are for subject 002(D). Top graphs compare Wxation in the left versus the right hemiWeld, summing scanning from eye and cheek regions. Subject 005(A) has lost the normal left hemiWeld bias, and subject 002(D) has an anomalous right hemiWeld bias. Bottom graphs compare Wxation in the upper (eyes and brow) versus the lower (nose, mouth, chin, and cheeks) face. Normal subjects Wxate the upper face more in upright faces and the lower face more in inverted faces. Subject 005(A) shows a similar pattern, but subject 002(D) Wxates the lower face more regardless of orientation faces, but the lower face more than the upper when the face was inverted (Fig. 5B). Furthermore, they tended to scan the side of the face in the left hemiWeld more than the side in the right hemiWeld, regardless of orientation (Fig. 5A). The comparison of subject 005(A) with the controls showed, for number of Wxations, a trend to a signiWcant overall diVerence in face parts scanned [F(1,441) = 1.96, P < 0.060], and a signiWcant interaction between group, 123 Exp Brain Res (2007) 181:199–211 orientation and face part [F(1,441) = 3.51, P < 0.0012). For scan time subject 005(A) diVered signiWcantly from the controls in face parts scanned [F(1,441) = 7.09, P < 0.0001], and a signiWcant interaction between group, orientation, and face part [F(1,441) = 4.67, P < 0.0001]. The general distribution of subject 005(A)’s scanning over facial regions was not that dissimilar to controls, though, with the main diVerence being that he did not show the normal bias for the left hemiWeld (Fig. 5). His division of Wxations between the upper versus lower face was very similar to the controls, with more upper than lower Wxations in upright faces and more lower than upper Wxations in inverted faces. The comparison of subject 002(D) with the controls showed a signiWcant diVerence in face parts scanned for both Wxation number [F(1,441) = 16.48, P < 0.0001] and scan time [F(1,441) = 13.15, P < 0.0001]. It also conWrmed a signiWcant interaction between group, face region and orientation [F(1,441) = 10.84, P < 0.0001]. Subject 002(D) had highly anomalous scan distribution, with an emphasis on regions that are not highly salient for controls, such as the left cheek and brow (Fig. 4). Her scanning was also highly skewed toward the right hemiWeld, and she did not show the normal concentration of the upper face and eye region in upright faces (Fig. 5). Scanning dynamics Markov and local repetitions In the controls, the compressed lambda showed an orientation eVect, with local scanning transitions being more random in inverted faces than in upright faces, but no eVect of fame. Comparisons of subject 005(A) and subject 002(D) with the controls showed no signiWcant diVerences. Subject 005(A) did show the normal inversion eVect for aLc, and subject 002(D) had a small inversion eVect for novel faces at least. For local repetitions, the controls show no signiWcant eVects of orientation or fame (Table 1). The comparisons of subject 005(A) and subject 002(D) with the controls showed that subject 005(A) had similar frequencies of local repetitions, while subject 002(D) had almost twice as many local repetitions as the controls [F(1,49) = 5.92, P < 0.046]. String editing costs Normal subjects show eVects of both fame and orientation, as well as a signiWcant interaction between the two. Editing costs are least for novel upright faces, suggesting that scanning is most stereotyped for these faces (Fig. 6). Subject 005(A) also showed a signiWcant eVect of fame [F(1,112) = 18.04, P < 0.0004], but no signiWcant eVect of orientation, or interaction with orientation. The eVect of 207 fame in subject 005(A) was anomalous, though, with less editing costs for famous than for novel faces. The comparison with normal subjects conWrmed a signiWcant diVerence in the eVect of fame [F(1,138) = 22.8, P < 0.0001]: while the controls had higher editing costs with famous than novel faces (t = 5.02, P < 0.0001), subject 005(A) had higher editing costs with novel faces than famous faces (t = 3.44, P < 0.0007). Linear contrasts between subject 005(A) and the controls showed that he had only slightly diVerent editing costs for famous faces (t = 2.02, P < 0.045) but much higher editing costs for novel faces (t = 5.04, P < 0.0001). Subject 002(D) showed no signiWcant eVects of fame or orientation. The comparison with the controls showed that subject 002(D) had much higher editing costs than the controls for all stimuli, regardless of fame or orientation [F(1,138) = 16.91, P < 0.007]. To summarize the temporal dynamics, the local structure of scanning (as indexed by the Markov and local repetition analyses) showed an orientation eVect in subject 005(A) and possibly a blunted one in subject 002(D); furthermore, subject 002(D) had an abnormally large number of local repetitions. Global scan structure, as indexed by the string editing analysis, was highly chaotic in subject 002(D) and not modulated by familiarity or orientation. The expertise eVect in subject 005(A) is also muted, in that he failed to show a signiWcant eVect of orientation. However, he had an anomalous familiarity eVect, with less stereotyped scanning of novel faces than famous faces, the reverse of the situation in the controls. This was due to the fact that his global scan patterns for novel faces were more random than those of controls, whereas his editing costs for famous faces were similar to those of controls. Discussion The results show signiWcant diVerences in face scan patterns between the prosopagnosics and the controls. Furthermore, Table 1 Local scanning transitions Control Mean SD DM KN 0.37 0.12 0.38 0.27 Asymmetric lambda Famous Upright Famous Inverted 0.31 0.12 0.24 0.30 Novel Upright 0.35 0.11 0.37 0.32 Novel Iinverted 0.26 0.07 0.21 0.28 0.25 0.10 0.34 0.40 Local repetitions Famous Upright Famous Inverted 0.23 0.07 0.35 0.45 Novel Upright 0.26 0.08 0.34 0.39 Novel Iinverted 0.23 0.06 0.30 0.46 123 208 Exp Brain Res (2007) 181:199–211 Fig. 6 Global scan patterns. String editing costs are shown against each of the morphed images in a series, ranging from 9 to 21, where 21 is 100% of the other face in a pair, and 11 is the 50:50 mixture. Control data at the top left show that editing costs are least for novel upright faces, suggesting a more stereotyped scanning of these faces. Error bars show one standard error for two of the series. At bottom left, subject 005(A) shows editing costs that are similar to controls for the famous upright faces, but higher editing costs than the controls for the novel upright faces. He also has a reduced inversion eVect for the novel faces. Subject 002(D) shows very high editing costs for all faces, without consistent eVects of orientation or fame they also show diVerences between the two prosopagnosics, suggesting that scan patterns might reXect not only eVects attributable to the absence of face recognition but also eVects related to the type of prosopagnosia. Besides the eVects of orientation and fame, which were the main foci of this report, a few general eVects emerged from the data. Face viewing is more diYcult for subjects 002(D) and 005(A) in general, as shown by their longer scan time, which was due in large part to longer Wxation durations. Hemianopia may have accentuated this diYculty in subject 005(A), but subject 002(D) has no such Weld defect. Normal subjects also showed a concentration of scanning in the left hemiWeld, regardless of orientation. However, subject 005(A) showed no hemiWeld asymmetry in scanning. This is not likely due to his hemianopia, as a left hemianopia should if anything lead to an adaptive increase in the scanning of the left hemiWeld, rather than a reduction (Behrmann et al. 1997; Barton et al. 1998). Subject 002(D), who does not have a hemianopia, has a pronounced bias to the opposite right hemiWeld. Previous studies of normal subjects have also shown left hemiWeld biases in at least some scanning parameters (Gallois et al. 1989; Mertens et al. 1993; Phillips and David 1997). Whether this hemiWeld asymmetry reXects a predominant role for the right hemisphere in processing faces has been argued (Grega et al. 1988), but the loss of the left bias in 123 subject 005(A), a patient with a visible right-sided lesion, is of interest in this debate. Evidence for a residual expert mechanism Is there evidence that subject 005(A), after a lesion of the right fusiform face area, has residual facial expertise? He does not show an orientation eVect on total scanning time or number of Wxations. While the controls scanned upright faces less, presumably because they are more eYcient at processing upright faces, subject 005(A) showed no diVerence between upright and inverted faces. However, the distribution of scanning by subject 005(A) is relatively normal and does show the normal pattern of a bias for the upper face in upright faces and a bias for the lower face in inverted faces. In scanning dynamics, transitions from one Wxation to another were less random in upright than in inverted faces, for both controls and subject 005(A). However, his global scan patterns showed only a trend to be more stereotyped for novel faces in the upright than inverted presentation, with no signiWcant orientation eVect. Furthermore, his global scan patterns with novel upright faces are more chaotic than those of the controls. Others have argued that the stereotyped scanning of novel faces reXects the guidance of a generic face schema possessed by the expert mechanism (Rizzo Exp Brain Res (2007) 181:199–211 et al. 1987). If so, subject 005(A)’s more chaotic scanning of novel faces may reXect degraded expert processing, consistent with his demonstrated diYculties in processing facial conWgurations. Along with the loss of the normal left hemiWeld bias, all of these Wndings suggest that subject 005(A)’s lesion has degraded but not completely eradicated all traces of his expert face processing mechanism. Subject 002(D), with developmental prosopagnosia, has more profound and anomalous changes. In terms of orientation eVects, she scans upright faces more than inverted ones, the opposite of normal. Her scanning of upright and inverted faces both concentrate on the lower face, whereas controls switch from emphasizing the upper face in upright stimuli to the lower face in inverted stimuli. She shows no eVect of orientation in her global scan patterns, which in controls are more chaotic for inverted faces. In other eVects, she has a highly anomalous distribution of scanning, spending more time on less salient regions such as the cheek and brow. She makes far more local repetitions of Wxations. Her global scan patterns are all highly chaotic. The overall impression is that subject 002(D) shows no evidence of normal face-expert mechanisms in her scanning of faces: she lacks normal orientation eVects, scans anomalous regions of the face, and has highly chaotic scan patterns. Thus subject 005(A) with an acquired unilateral lesion appears to have some residual face-expertise mechanisms, while subject 002(D) with developmental prosopagnosia has virtually none. The relatively preserved distribution of subject 005(A)’s Wxations is consistent with conclusions of normal Wxation distribution in another patient with acquired prosopagnosia (Le et al. 2003). In contrast, subject 002(D) has particularly chaotic scanning, with disorganized global sequences and anomalous distributions of Wxations that do not mirror the normal distribution of regional saliency within faces (Shepherd et al. 1981). This too, is similar to a previous study of four patients with hereditary prosopagnosia, which showed a tendency of these subjects to Wxate external facial elements rather than the internal features (Schwarzer et al. 2006). Subject 002(D)’s de-emphasis of the eye region and increased scanning of less salient brow and cheek regions is also reminiscent of other reports of anomalous scanning in other patient groups. These include the Wnding that autistic subjects Wxate the mouth more than the eyes (Klin et al. 2002), and that schizophrenic subjects Wxate salient facial features less (Loughland et al. 2002). The data of subject 002(D) raise the possibility that these other Wndings might reXect an innate disorganization of face processing. However, caution is required for such claims, as there are other possible explanations for such anomalous scan patterns, including avoidance of social engagement, for example. 209 Evidence for residual covert familiarity Fame eVects would indicate the existence of residual facial memories and their ability to inXuence data acquisition during face viewing. This residual ability may be considered a type of covert face recognition in that, while prosopagnosic patients deny recognizing faces, inXuences of facial familiarity can still be found in their behavior or perceptual decisions (Bruyer 1991; Young 1994). Subject 005(A) showed a trend to the normal eVect of fewer Wxations on famous faces, but tended to take more time with each Wxation on upright famous faces, an anomalous eVect. His global scan patterns also showed a signiWcant diVerent between famous and novel faces, but this too was anomalous, in that novel faces were more chaotically scanned than famous faces. However, this was due to novel faces being scanned more randomly by subject 005(A) than by controls, whereas his global scan patterns for famous faces were comparable to controls. We suggest that these results might indicate that subject 005(A)’s scanning of famous faces are still guided by residual facial memories. Previous studies have shown that he has reasonably good imagery for famous faces, suggesting that his facial memories may be relatively spared (Barton and Cherkasova 2003). This contrasts with his more chaotic scanning of novel faces, which may reXect impaired perceptual processing and access to a generic facial schema. Subject 002(D) showed fewer familiarity eVects than subject 005(A). She did spend less overall scan time viewing famous faces, as did normal controls. She also made shorter Wxations on famous faces, whereas normal subjects did not show an eVect of fame on the duration of Wxations. Of note, subject 002(D) showed no signiWcant eVect of fame in her global scan patterns. Thus the eVects of fame are more prominent in subject 005(A) than in subject 002(D). Subject 005(A) has global scan patterns for famous faces that are similar in their degree of stereotypy to those of controls, and less chaotic than his scanning of novel faces. This suggests that the presence of facial memories in subject 005(A) can organize his scanning of famous faces in a manner similar to controls, even though his scanning of novel faces is more disorganized. However, subject 005(A) also diVers from controls in that his Wxations on famous faces last longer. Longer Wxations can be found when perceptual information is degraded artiWcially (van Diepen 1995). In subject 005(A), it may be that weak matches between his pathologically degraded percepts and intact facial memories generate a drive to increase data acquisition time during each Wxation, more than occurs for novel faces where no strong memories exist. These Wndings provide interesting parallels to our other behavioral data on covert recognition. Patients with unilateral 123 210 right occipitotemporal lesions had the strongest evidence of covert recognition with either direct forced-choice or indirect priming techniques (Barton et al. 2001a, 2004). These covert eVects correlated with their residual imagery for faces, which was also best preserved in patients with right occipitotemporal lesions (Barton and Cherkasova 2003). In contrast, patients with childhood-onset prosopagnosia had little covert recognition or facial imagery (but see (Michelon and Biederman 2003) for a report of a developmental prosopagnosic with residual face imagery). We postulated Wrst, that residual facial memories may be a prerequisite for covert recognition, and second, that these may be lacking in childhood-onset prosopagnosia, as any failure in the complex hierarchy of processes in face recognition would preclude the establishment of facial memories after early-onset lesions. In this report we Wnd stronger eVects of fame on scan patterns in a subject with unilateral right occipitotemporal lesion, with good evidence of covert ability and facial imagery in our prior reports, and weaker fame eVects (chieXy a diVerence in total scan time) in the patient with developmental prosopagnosia. In summary, investigations of the scan patterns in these two patients reveal degradation of expert face processing in both patients, in whom prior testing had shown severe impairments in the perceptual processing of facial conWguration. This degradation was more profound in subject 002(D), who had a developmental prosopagnosia. However, residual eVects of familiarity were still evident in the global scan patterns of subject 005(A), who has evidence of good facial imagery and covert recognition, but not in subject 002(D). These Wndings show how the distribution and organization of scanning Wxations can yield data about the information processing of faces in prosopagnosic patients. Acknowledgments JB was supported by a Canada Research Chair, a Michael Smith Foundation for Health Research Senior Scholarship, CIHR MOP-77615 and NIMH R01-MH069898. James Intriligator assisted with experimental programming and Rebecca Hefter with data analysis. References AlthoV R, Cohen N (1999) Eye-movement-based memory eVect: a reprocessing eVect in face perception. J Exp Psychol Learn Mem Cogn 25:997–1010 Bartlett J, Searcy J (1993) Inversion and conWguration of faces. Cognit Psychol 25:281–316 Barton J, Cherkasova M (2003) Face imagery and its relation to perception and covert recognition in prosopagnosia. Neurology 61:220–225 Barton J, Behrmann M, Black S (1998) Ocular search during line bisection. The eVects of hemi-neglect and hemianopia. Brain 121:1117–1131 123 Exp Brain Res (2007) 181:199–211 Barton J, Cherkasova M, O’Connor M (2001a) Covert recognition in acquired and developmental prosopagnosia. Neurology 57:1161– 1167 Barton J, Keenan J, Bass T (2001b) Discrimination of spatial relations and features in faces: eVects of inversion and viewing duration. Brit J Psychol 92:527–549 Barton J, Press D, Keenan J, O’Connor M (2002) Lesions of the fusiform face area impair perception of facial conWguration in prosopagnosia. Neurology 58:71–78 Barton J, Cherkasova M, Press D, Intriligator J, O’Connor M (2003a) Developmental prosopagnosia: a study of three patients. Brain Cogn 51:12–30 Barton J, Deepak S, Malik N (2003b) Attending to faces: change detection, familiarisation and inversion eVects. Perception 32:15–28 Barton J, Cherkasova M, Hefter R (2004) The covert priming eVect of faces in prosopagnosia. Neurology 63:2062–2068 Barton J, RadcliVe N, Cherkasova M, Edelman J, Intriligator J (2006) Information processing during face recognition: the eVects of familiarity, inversion and morphing on scanning Wxations. Perception 353:1089–1105 Behrmann M, Watt S, Black S, Barton J (1997) Impaired visual search in patients with unilateral neglect: an oculographic analysis. Neuropsychologia 35:1445–1458 Brandt S, Stark L (1997) Spontaneous eye movements during visual imagery reXect the content of the visual scene. J Cogn Neurosci 9:27–38 Bruyer R (1991) Covert facial recognition in prosopagnosia: a review. Brain Cogn 15:223–235 Deco G, Schurmann B (2000) A neuro-cognitive visual system for object recognition based on testing of interactive attentional topdown hypotheses. Perception 29:1249–1264 Farah M, Wilson K, Drain M, Tanaka J (1998) What is “special” about face perception? Psychol Rev 105:482–498 Gallois P, Buquet C, Charlier J, Paris V, Hache J-C, Dereux J-F (1989) Asymmetry in the strategy for visual perception of faces and emotional facial expression. Rev Neurol 145:661–664 Gbadamosi J, Zangemeister WH (2001) Visual imagery in hemianopic patients. J Cogn Neurosci 13:855–866 Grega D, Sackheim H, Sanchez E, Cohen B, Hough S (1988) Perceiver bias in the processing of human faces: neuropsychological mechanisms. Cortex 24:91–117 Groner R, Walder F, Groner M (1984) Looking at faces: local and global aspects of scanpaths. In: Gale A, Johnson F (eds) Theoretical and applied aspects of eye movement research. Elsevier, Amsterdam, pp 523–533 Henderson J (2003) Human gaze control during real-world scene perception. Trends Cogn Sci 7:498–504 Kemp R, McManus C, Pigott T (1990) Sensitivity to the displacement of facial features in negative and inverted images. Perception 19:531–543 Klin A, Jones W, Schultz R, Volkmar F, Cohen D (2002) Visual Wxation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Arch Gen Psychiatry 59:809–816 Le S, Raufaste E, Demonet JF (2003) Processing of normal, inverted, and scrambled faces in a patient with prosopagnosia: behavioural and eye tracking data. Brain Res Cogn Brain Res 17:26–35 Leder H, Bruce V (1998) Local and relational aspects of face distinctiveness. Q J Exp Psychol A 51:449–473 Loughland CM, Williams LM, Gordon E (2002) Visual scanpaths to positive and negative facial emotions in an outpatient schizophrenia sample. Schizophr Res 55:159–170 Malcolm G, Leung C, Barton J (2004) Regional variation in the inversion eVect for faces: diVerent patterns for feature shape, spatial relations, and external contour. Perception 33:1221–1231 Exp Brain Res (2007) 181:199–211 Mertens I, Siegmund H, Grusser OJ (1993) Gaze motor asymmetries in the perception of faces during a memory task. Neuropsychologia 31:989–998 Michelon P, Biederman I (2003) Less impairment in face imagery than face perception in early prosopagnosia. Neuropsychologia 41:421–441 Phillips M, David A (1997) Viewing strategies for simple and chimeric faces: an investigations of perceptual bias in normals and schizophrenic patients using visual scan paths. Brain Cogn 35:225–238 Rizzo M, Hurtig R, Damasio A (1987) The role of scanpaths in facial recognition and learning. Ann Neurol 22:41–45 Rybak I, Gusakova V, Golovan A, Podladchikova L, Shevtsova N (1998) A model of attention-guided visual perception and recognition. Vision Res 38:2387–2400 Schwarzer G, Huber S, Gruter M, Gruter T, Gross C, Hipfel M, Kennerknecht I (2006) Gaze behaviour in hereditary prosopagnosia. Psychol Res [epub] Searcy J, Bartlett J (1996) Inversion and processing of component and spatial-relational information in faces. J Exp Psychol 22:904–915 211 Sergent J (1984) An investigation into component and conWgurational processes underlying face perception. Brit J Psychol 75:221–242 Shepherd J, Davies G, Ellis H (1981) Studies of cue saliency. In: Davies G, Ellis H, Shepherd J (eds) Perceiving and remembering faces. Academic, London, pp 105–131 Stacey P, Walker S, Underwood J (2005) Face processing and familiarity: evidence from eye-movement data. Brit J Psychol 96:407–422 van Diepen P (1995) Chronometry of foveal information extraction during scene perception. In: Findlay J, Walker R, Kentridge R (eds) Eye movement research: mechanisms, processes and applications. Elsevier, Amsterdam, pp 349–362 Yin R (1970) Face recognition: a dissociable ability? Neuropsychologia 23:395–402 Young A (1994) Covert recognition. In: Farah M, RatcliV G (eds) The neuropsychology of high-level vision. Lawrence Erlbaum Associates, Hillsdale, pp 331–358 Zangemeister W, Sherman K, Stark L (1995) Evidence for a global scanpath strategy in viewing abstract compared with realistic images. Neuropsychologia 33:1009–1025 123