Beyond Siri: Researchers are Bridging Human-Computer Interaction

Scientists are developing new technologies for making computers recognize not just traditional commands, but also non-verbal ones

Written byColorado State University

| 2 min read

Listen with Speechify

0:00

2:00

researchers are developing new technologies for making computers recognize not just traditional commands, but also non-verbal ones Photo courtesy of Colorado State UniversityFor most people, using a computer is limited to clicking, typing, searching, and, thanks to Siri and similar software, verbal commands.

Compare that with how humans interact with each other, face to face – smiling, frowning, pointing, tone of voice all lend richness to communication.

With the goal of revolutionizing everyday interactions between humans and computers, Colorado State University researchers are developing new technologies for making computers recognize not just traditional commands, but also non-verbal ones – gestures, body language and facial expressions.

Communicating through gestures

Their project, titled “Communication Through Gestures, Expression and Shared Perception,” is led by Professor of Computer Science Bruce Draper, and is bolstered by a recent $2.1 million grant from the Defense Advanced Research Projects Agency (DARPA) under its “Communicating with Computers” funding program.

“Current human-computer interfaces are still severely limited,” said Draper, who is joined on the project by CSU researchers from the computer science and mathematics departments. “First, they provide essentially one-way communication: users tell the computer what to do. This was fine when computers were crude tools, but more and more, computers are becoming our partners and assistants in complex tasks. Communication with computers needs to become a two-way dialogue.”

Packets of gesture info

The team has proposed creating a library of what are called Elementary Composable Ideas (ECIs). Like little packets of information recognizable to computers, each ECI contains information about a gesture or facial expression, derived from human users, as well as a syntactical element that constrains how the information can be read.

To achieve this, the researchers have set up a Microsoft Kinect interface. A human subject sits down at a table with blocks, pictures and other stimuli. The researchers try to communicate with and record the person’s natural gestures for concepts like “stop,” or, “huh?”

“We don’t want to say what gestures you should use,” Draper explained. “We want people to come in and tell us what gestures are natural. Then, we take those gestures and say, ‘OK, if that’s a natural gesture, how do we recognize it in real time, and what are its semantics? What roles does it play in the conversation? When do you use it? When do you not use it?’”

Their goal: making computers smart enough to reliably recognize non-verbal cues from humans in the most natural, intuitive way possible. According to the project proposal, the work could someday allow people to communicate more easily with computers in noisy settings, or when a person is deaf or hard of hearing, or speaks another language.

DARPA basic research

The project, which falls broadly under DARPA’s basic research arm, is focused on enabling people to talk to computers through gestures and expressions in addition to words, not in place of them, the researchers say.

The project includes co-principal investigators Ross Beveridge, professor of computer science; Jaime Ruiz, assistant professor of computer science; and Michael Kirby and Chris Peterson, both professors of mathematics.

For most people, using a computer is limited to clicking, typing, searching, and, thanks to Siri and similar software, verbal commands.

Compare that with how humans interact with each other, face to face – smiling, frowning, pointing, tone of voice all lend richness to communication.