How We Read Faces

Chapter 7

How We Read Faces

A dedicated piece of brain

Somewhere in the fusiform gyrus, on the underside of the temporal lobe, sits a patch of cortex that lights up almost exclusively when you look at a face. In 1997, the cognitive neuroscientist Nancy Kanwisher and colleagues used functional MRI to map it directly: in most subjects tested, a specific region — since named the fusiform face area, or FFA — responded far more strongly to faces than to houses, hands, or scrambled versions of the same face images, and the effect held up against explanations based on attention or generic object recognition.¹ Damage to this region, or the circuitry feeding it, produces prosopagnosia — face blindness — a startlingly specific deficit in which people can still recognize a chair, a dog, or a friend's voice, but lose the ability to recognize faces, sometimes including their own in a mirror, while every other kind of visual object recognition stays intact. Face perception — knowing that a pattern of light is a face, and whose face it is — is fast, largely automatic, and runs on dedicated neural hardware that a lifetime of ordinary object recognition doesn't require.

A famous low-tech demonstration makes the same point without a scanner. In 1980 the psychologist Peter Thompson took a photo of Margaret Thatcher and flipped just the eyes and mouth; turned upright the result looks grossly wrong, but with the whole face inverted most viewers barely notice the distortion.² The effect — the "Thatcher illusion" — points to the same conclusion as the brain imaging: face processing is specialized and orientation-dependent in a way ordinary object recognition is not.

What makes a face attractive

Face reading — deciding what a face means — is a different kind of process, and it starts with attractiveness. In 1990, the psychologists Judith Langlois and Lori Roggman digitally averaged photographs of college students' faces into composites, and found that the composites were consistently rated more attractive than almost all of the individual faces that went into them, with attractiveness climbing as more faces were blended in.³ The finding wasn't a quirk of one undergraduate population: a later study of the Hadza, an isolated hunter-gatherer community in Tanzania with little exposure to Western media, found the same preference for facial averageness holding within their own population, even as cross-cultural face preferences broke down elsewhere — evidence that the pull toward the average face isn't simply a learned aesthetic.⁴ Randy Thornhill and Steven Gangestad's parallel line of research pointed to a second ingredient: facial and bodily symmetry. Small, random deviations from perfect bilateral symmetry — fluctuating asymmetry — tend to increase with developmental stress and poor health during growth, and Thornhill and Gangestad argued that low fluctuating asymmetry and averageness both function as signals of underlying genetic quality — resistance to parasites and infection — a "good genes" hypothesis that has since drawn substantial skepticism and mixed replication.⁵ David Perrett's morphing experiments pushed the symmetry side further, digitally warping ordinary faces toward greater symmetry and showing the manipulated versions reliably out-rated the originals — evidence that symmetry doesn't just correlate with attractiveness but can causally shift it.⁶ His work on averageness told a more complicated story: a composite built only from faces already rated attractive out-rated a plain average composite, and exaggerating that departure from the average raised attractiveness further — a result offered as evidence against the simple idea that the average face is the most attractive one.⁷

The preference shows up strikingly early. In a 1987 study, Langlois and colleagues showed 2-to-3-month-old infants pairs of adult women's faces that adults had rated attractive or plain, and the babies looked longer at the attractive ones — a pull toward certain faces already in place before much exposure to media or culture.⁸ It doesn't settle whether the preference is innate or very rapidly learned, but it makes a purely cultural account of facial beauty harder to sustain.

The unreliable inference

Attractiveness is where the reading stops being reliable. People don't just rate faces as attractive or plain — they read trustworthiness, competence, intelligence, and dominance into a face within a fraction of a second of seeing it, and those snap judgments are remarkably consistent from one observer to the next: shown the same unfamiliar face, different people tend to agree on who looks "trustworthy" or "competent." The trouble is that agreement is not accuracy. Decades of research comparing these first-impression ratings against the actual traits, behavior, and honesty of the people being judged has found the correlation between how trustworthy a face looks and how trustworthy the person actually is to be weak at best and often statistically indistinguishable from zero. The brain is running a real, fast, automatic inference engine on faces — it's just an engine built to be consistent with itself, not to be correct about the people wearing the faces it's reading.

One study makes the stakes of that unreliability concrete. In 2005 the psychologist Alexander Todorov showed people pairs of election-candidate photos and asked, in about a second, which face looked more competent; that snap judgment alone predicted the actual winner in roughly two-thirds of U.S. Senate races — no policies, no party labels, just a face.⁹ The judgment was fast, confident, and widely shared across observers, and it still wasn't a reliable guide to who should govern.

The uncanny valley

Face reading also has an odd failure mode at its far edge. In 1970, the Japanese roboticist Masahiro Mori proposed that as an artificial face or figure is made more humanlike, people's emotional response rises with it — up to a point where near-perfect but not-quite-right human likeness produces a sharp plunge into unease and revulsion, before recovering only once the likeness becomes indistinguishable from a real person.¹⁰ Mori called the dip bukimi no tani — the uncanny valley — and it has since become a standard reference point in robotics, animation, and game design for why almost-human faces so often read as unsettling rather than charming.

Perceiving versus reading

Put the two halves of this chapter side by side and a single distinction falls out. Perceiving a face — is this a face, and whose is it — is one of the best-established, most specialized abilities in human cognition, wired into a specific patch of cortex and disrupted in strikingly selective ways when that circuitry fails. Reading a face — what kind of person is behind it — is just as fast and just as automatic, but it is not equally trustworthy; it is a set of intuitive heuristics about averageness, symmetry, and expression standing in for judgments about character that the face alone cannot reliably deliver. That gap between the confidence of the impression and the accuracy of the inference is exactly what made physiognomy so seductive in earlier eras of this tour — the instinct to read a face is fast, vivid, and shared across observers, which feels like evidence of truth. Modern cognitive science suggests it's mostly evidence of how the reading machinery works, not of what it's reading.

Further reading