To the unfamiliar, it might seem a Jedi mind trick – the notion of controlling robots by simply thinking of what you wish for them to do. If so, then Joe Francis, associate professor of biomedical engineering, is Obi-Wan Kenobi. The force, along with a new grant from the National Science Foundation (NSF) for $308,077, is with him as he works to expand humans’ ability to work with robots.
To Francis, his project, “NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction,” is all about teamwork. “We’re teaming the human with the robotic systems,” he said. “The robots can do certain things really well, but they’re going to need some guidance, and rather than have the person guide them continuously with trying to inform them about everything, there’s this interplay between the two systems.”
The goal is to determine how well artificial agents and humans can work together instead of getting in each other’s way.
It’s personal
Once that teamwork is improved, Francis believes it and other research he is conducting will change people’s lives by improving sensory prosthetics. His devotion to this belief is a deeply-felt cause, one he’s probably had since first grade.
That was the year he lost the big toe on his left foot in a terrible accident, and he immediately found ways to overcome the loss, excelling at sports quickly and all the way through college. Sure enough, he helped himself profoundly with his determination. Now he’s set his mind on doing the same for others.
“If somebody is blind, I want them to be able to see. If somebody can’t hear, I want them to be able to hear. If they don’t have an arm, I want to give them an arm and be able to control it naturally and I want them to feel it as if it was a part of them so they don’t feel like they’re using this tool, but they feel whole again,” said Francis.
That’s where focus shifts to the brain. Current human robot interaction (HRI) centers around video, keyboards and, plainly, just speaking to them – telling them what you want. The world Francis envisions makes robots part of everyday life and our communication with them second nature. That’s also the idea of co-robotics, as expressed by the NSF, which acknowledges humans as part of a larger robotic control system.
Coordinating the team
Francis works to take information from humans and give it to robots so they can anticipate what humans want in the future. He does this by recording human brain waves through EEG and brain hemodynamics, or blood changes, through an imaging process called Functional Near-Infrared Spectroscopy (fNIRS).
“From that we can determine how active different brain regions are,” said Francis. He also collects data by watching the eyes and recording them with high-speed cameras since pupils actually change size when certain rewarding or punishing events happen.
Unexpectedly, the researchers can also determine a reward/expectation signal from a person’s brain.
“We can basically tell if they think something is leading to a reward or something good,” said Francis.
Once that human data is collected, Francis takes it to improve the software for his robots and improve their learning agent.
He’s robotic, too
In the old television series, “The Six Million Dollar Man,” the lead character had robotic and bionic limbs. Francis doesn’t have any of that, but you can call him the six million dollar man, too, because his robot research has brought in just about that amount of funding since arriving at the UH Cullen College of Engineering in March.
In the words of Obi-Wan, “I have the high ground."