University of Houston Cullen College of Engineering


Professor Joe Francis’ Mind is Teeming with Ways to Team with Robots


Laurie Fickman
In a galaxy on UH campus, Professor Joe Francis is perfecting Jedi mind control
In a galaxy on UH campus, Professor Joe Francis is perfecting Jedi mind control

To the unfamiliar, it might seem a Jedi mind trick – the notion of controlling robots by simply thinking of what you wish for them to do. If so, then Joe Francis, associate professor of biomedical engineering, is Obi-Wan Kenobi. The force, along with a new grant from the National Science Foundation (NSF) for $308,077, is with him as he works to expand humans’ ability to work with robots.

To Francis, his project, “NRI: Collaborative Research: Multimodal Brain Computer Interface for Human-Robot Interaction,” is all about teamwork. “We’re teaming the human with the robotic systems,” he said. “The robots can do certain things really well, but they’re going to need some guidance, and rather than have the person guide them continuously with trying to inform them about everything, there’s this interplay between the two systems.”

The goal is to determine how well artificial agents and humans can work together instead of getting in each other’s way.

It’s personal

Once that teamwork is improved, Francis believes it and other research he is conducting will change people’s lives by improving sensory prosthetics. His devotion to this belief is a deeply-felt cause, one he’s probably had since first grade.

That was the year he lost the big toe on his left foot in a terrible accident, and he immediately found ways to overcome the loss, excelling at sports quickly and all the way through college. Sure enough, he helped himself profoundly with his determination. Now he’s set his mind on doing the same for others.

“If somebody is blind, I want them to be able to see. If somebody can’t hear, I want them to be able to hear. If they don’t have an arm, I want to give them an arm and be able to control it naturally and I want them to feel it as if it was a part of them so they don’t feel like they’re using this tool, but they feel whole again,” said Francis.

That’s where focus shifts to the brain. Current human robot interaction (HRI) centers around video, keyboards and, plainly, just speaking to them – telling them what you want. The world Francis envisions makes robots part of everyday life and our communication with them second nature. That’s also the idea of co-robotics, as expressed by the NSF, which acknowledges humans as part of a larger robotic control system.

Coordinating the team

Francis works to take information from humans and give it to robots so they can anticipate what humans want in the future. He does this by recording human brain waves through EEG and brain hemodynamics, or blood changes, through an imaging process called Functional Near-Infrared Spectroscopy (fNIRS).

“From that we can determine how active different brain regions are,” said Francis. He also collects data by watching the eyes and recording them with high-speed cameras since pupils actually change size when certain rewarding or punishing events happen.

Unexpectedly, the researchers can also determine a reward/expectation signal from a person’s brain.

“We can basically tell if they think something is leading to a reward or something good,” said Francis.

Once that human data is collected, Francis takes it to improve the software for his robots and improve their learning agent.

He’s robotic, too

In the old television series, “The Six Million Dollar Man,” the lead character had robotic and bionic limbs. Francis doesn’t have any of that, but you can call him the six million dollar man, too, because his robot research has brought in just about that amount of funding since arriving at the UH Cullen College of Engineering in March.

In the words of Obi-Wan, “I have the high ground."



Related News Stories

Understanding Congenital Heart Defects To Prevent Them

Determining how hearts develop in utero is critical to understanding congenital heart defects.

UH Engineer Using Optical Equipment to Watch Heart Develop


To understand cardiovascular failures, the leading cause of birth defect-related deaths in infants, UH professor of biomedical engineering Kirill Larin is teaming up with Baylor College of Medicine professor of cellular and molecular physiology Irina Larina on a chicken and egg hunt.

Distinguished Alumnus, Bilfinger Executive to be UH Cullen College Fall 2018 Commencement Speaker

Terrance “Terri” Ivers, P.E. (BSME ’80) is the fall 2018 UH Cullen College of Engineering commencement speaker.

Terrance “Terri” Ivers, P.E. (BSME ’80), is coming home to the UH Cullen College of Engineering as the featured speaker at its commencement on Dec. 13 at the NRG Arena. He received his Bachelor of Science degree in mechanical engineering from UH in 1980 and has remained involved in the Cullen College community throughout the course of his 37 year-long career.

Renowned Engineering Faculty Join Cullen College

Birol Dindoruk, Professor of Petroleum Engineering at UH Cullen College

The UH Cullen College of Engineering is kicking off the 2018-2019 academic year by adding seven new faculty members to its teaching and research rosters, including a member of the presigious National Academy of Engineering. They specialize in fields ranging from reservoir engineering to neuro-engineering, and their research covers topics ranging from water resources to power and energy systems.