University of Houston Cullen College of Engineering


Professor Joe Francis’ Mind is Teeming with Ways to Team with Robots


Laurie Fickman
In a galaxy on UH campus, Professor Joe Francis is perfecting Jedi mind control
In a galaxy on UH campus, Professor Joe Francis is perfecting Jedi mind control

To the unfamiliar, it might seem a Jedi mind trick – the notion of controlling robots by simply thinking of what you wish for them to do. If so, then Joe Francis, associate professor of biomedical engineering, is Obi-Wan Kenobi. The force, along with a new grant from the National Science Foundation (NSF) for $308,077, is with him as he works to expand humans’ ability to work with robots.

To Francis, his project, “NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction,” is all about teamwork. “We’re teaming the human with the robotic systems,” he said. “The robots can do certain things really well, but they’re going to need some guidance, and rather than have the person guide them continuously with trying to inform them about everything, there’s this interplay between the two systems.”

The goal is to determine how well artificial agents and humans can work together instead of getting in each other’s way.

It’s personal

Once that teamwork is improved, Francis believes it and other research he is conducting will change people’s lives by improving sensory prosthetics. His devotion to this belief is a deeply-felt cause, one he’s probably had since first grade.

That was the year he lost the big toe on his left foot in a terrible accident, and he immediately found ways to overcome the loss, excelling at sports quickly and all the way through college. Sure enough, he helped himself profoundly with his determination. Now he’s set his mind on doing the same for others.

“If somebody is blind, I want them to be able to see. If somebody can’t hear, I want them to be able to hear. If they don’t have an arm, I want to give them an arm and be able to control it naturally and I want them to feel it as if it was a part of them so they don’t feel like they’re using this tool, but they feel whole again,” said Francis.

That’s where focus shifts to the brain. Current human robot interaction (HRI) centers around video, keyboards and, plainly, just speaking to them – telling them what you want. The world Francis envisions makes robots part of everyday life and our communication with them second nature. That’s also the idea of co-robotics, as expressed by the NSF, which acknowledges humans as part of a larger robotic control system.

Coordinating the team

Francis works to take information from humans and give it to robots so they can anticipate what humans want in the future. He does this by recording human brain waves through EEG and brain hemodynamics, or blood changes, through an imaging process called Functional Near-Infrared Spectroscopy (fNIRS).

“From that we can determine how active different brain regions are,” said Francis. He also collects data by watching the eyes and recording them with high-speed cameras since pupils actually change size when certain rewarding or punishing events happen.

Unexpectedly, the researchers can also determine a reward/expectation signal from a person’s brain.

“We can basically tell if they think something is leading to a reward or something good,” said Francis.

Once that human data is collected, Francis takes it to improve the software for his robots and improve their learning agent.

He’s robotic, too

In the old television series, “The Six Million Dollar Man,” the lead character had robotic and bionic limbs. Francis doesn’t have any of that, but you can call him the six million dollar man, too, because his robot research has brought in just about that amount of funding since arriving at the UH Cullen College of Engineering in March.

In the words of Obi-Wan, “I have the high ground."



Related News Stories

Cullen College Professor Wins Outstanding Young Alumni Award

Jerrod Henderson, instructional associate professor at the UH Cullen College of Engineering, with some of his students.

Jerrod Henderson Recognized for Diversity and Student Success Efforts


Jerrod Henderson, instructional associate professor at the UH Cullen College of Engineering, will receive the 2019 Outstanding Young Alumni Award from his alma mater. The College of Liberal Arts and Sciences at the University of Illinois chose seven alumni to recognize.

Testing New Treatment for Epilepsy Patients

Location of the seizure onset zone varies in each patient. The procedure to locate it begins with surgical insertion of electrodes.

Seizure Onset Zone Located Immediately, Dealt With


University of Houston associate professor of biomedical engineering Nuri Ince, who pioneered a dramatic decrease in the time it takes to detect the seizure onset zone (SOZ) in the brain, has been awarded $2.3 million by the National Institutes of Health to expand his testing in a large number of adult and pediatric epilepsy cases.