RIT faculty earns NSF CAREER award to study human behavior using machine learning

A Rochester Institute of Technology professor has earned a prestigious National Science Foundation award to use computers to better understand human behavior and social interaction.

Ifeoma Nwogu, an assistant professor of computer science, received an NSF Faculty Early Career Development (CAREER) award and grant for her five-year project.

She aims to study human behavior in a new way, by using machine learning techniques to analyze and find patterns in the many signals that individuals display during social interactions. Her work will specifically look at groups working in science, technology, engineering and math (STEM), with the aim of supporting underrepresented groups in STEM.

“In a conversation, people are constantly displaying and processing different non-verbal signals, such as how fast someone is talking or the facial expressions they are making,” said Nwogu. “I wanted to use computers to analyze all of these signals in new ways and help us gain a better understanding of social intelligence and our emotions.”

Working with a team of student researchers, Nwogu will observe and collect data generated in small face-to-face group meetings. Some of the non-verbal communication data she will collect include galvanic skin response information, facial expressions, gestures, self-observed data and speech prosody—which includes intonation, tone, stress and rhythm of speech.

The sequence of data is essentially time series signals, and the real challenge is extracting meaningful statistics and analyzing patterns in that data, Nwogu explained. Using Bayesian trained neural networks, she plans to develop a computational framework for analyzing individuals’ behavior and interactions.

“Deep learning is good at analyzing large data sets and finding hidden patterns, layers or clusters, but it can’t tell you why there are patterns,” Nwogu said. “We can then create probabilistic models to help gain a better understanding of collective behaviors in a group, such as why people with a certain perceived-personality type might speak at a specific volume or pace.”

For example, Nwogu could use the models to predict if people in a group have rapport or not. Speech volume or certain facial expressions could be indicators used for making that prediction.

For the project, Nwogu is partnering with RIT’s Center for Advancing STEM Teaching, Learning and Evaluation (CASTLE), which is a network of faculty, projects and programs engaged in scholarship surrounding STEM education. 

Nwogu will observe and collect signal data from student and faculty mentoring or peer-to-peer groups that the center offers. She hopes to better understand everything that happens, to find ways to improve teaching in STEM. For example, the study may observe what happens when a student becomes flustered by a difficult topic or when an instructor becomes frustrated teaching a specific concept.

“Ifeoma’s work has the potential to truly revolutionize how we study student learning,” said Scott Franklin, director of CASTLE and a professor of physics. “Social and group interactions are critical to learning, which is why activity-based classrooms are so successful, but research on human dynamics, which are incredibly subtle, has always been extremely labor-intensive. Machine-learning can automate procedures that reliably identify complicated group dynamics, exponentially increasing our ability to acquire and analyze important data on how students work together to construct knowledge and learn content.”

As part of outreach for her research, Nwogu recently taught a computer vision course at the African Institute of Mathematical Sciences (AIMS) in Rwanda.

As part of outreach for the study, Nwogu has partnered with RIT’s African Center for Excellence in Teaching and Learning Mathematics and Science to teach a course at the African Institute of Mathematical Sciences (AIMS) in Rwanda. She also plans to bring research students to Rwanda and continue the NSF project, observing if social interaction patterns change across different cultures.

The CAREER program is an NSF-wide activity that offers awards in support of junior faculty who exemplify the role of teacher-scholars through outstanding research, excellent education and the integration of education and research within the context of the mission of their organizations. 

“Congratulations to Ifeoma on this tremendous achievement,” said Anne Haake, dean of RIT’s Golisano College of Computing and Information Sciences. “NSF CAREER awards like this are a testament to the increasing strength of computing research by our faculty at RIT.”