RIT/NTID researchers study how deaf and hearing people watch sign language

A recent study has shown that readers’ eye gaze behaviors are strong indicators of words that are unexpected, new, or difficult to understand. The study by Rain Bosworth, an assistant professor and researcher in the Center for Sensory, Perceptual, and Cognitive Ecology (SPaCE Center) at Rochester Institute of Technology’s National Technical Institute for the Deaf, explores the unknown qualities of gaze behavior for “sign watching” and how these are affected by a user’s language expertise and intelligibility of the sign input.  

According to Bosworth’s study, published in the Journal of Deaf Studies and Deaf Education, with NTID graduate Adam Stone, gaze behaviors can provide an index of cognitive effort and knowledge in signers. This study provides the first evidence that novice and fluent signers have different eye gaze behaviors.  

Bosworth and her team recorded gaze behaviors in 52 deaf and hearing adults while they watched signed narratives. Highly fluent signers primarily kept a steady gaze on the face and used peripheral vision to perceive the signers’ moving hands. The researchers then showed the participants videos of signed stories played backwards. Bosworth said that people who learned American Sign Language earlier in life are better equipped to understand difficult video-reversed narratives. Fluent signers tended to focus strongly on the face when sign watching, even for low intelligibility conditions.

“These low intelligibility conditions simulate what happens in real-world settings when trying to watch live signers on phones with small displays or with weak internet signals,” explained Bosworth.

Novice signers, who scored lower on measures of story comprehension, showed a very different gaze pattern.

“Gaze behavior is more scattered for people who recently learned sign language, and this scatter increased for low-intelligibility conditions, probably because observers are looking directly at the moving hands,” Bosworth said. “This fits with what we know about research that shows that signers have very good peripheral vision, especially from the lower visual field.  Expert signers look at the face and utilize their peripheral vision for catching the fine details of moving handshapes.” 

But, there is some good news for non-signers. According to Bosworth, it doesn’t take long for signers to develop “expert-like” gaze patterns during sign comprehension. Hearing signers who have been signing for at least five years often show steady gaze behavior on the face just like fluent deaf signers.

A video illustration of the results is available.