Bailey Russ ’18 and Frankie Capraro ’18 are laying the foundation for an Xbox program that could change how physical therapy patients recover at home.
Currently there are games available for Xbox that involve virtual workouts, but none that involve coaching. The Xbox’s Kinect camera can detect players’ motions, but without accountability, the games’ energetic commands ultimately amount to an exercise video—not necessarily at-home therapy.
But what if the game could “see” that a patient performed only four bicep curls, when their regiment asks for eight, and encouraged them to finish? Russ, a kinesiology major with an emphasis in physical therapy, can envision how such an artificial intelligence program would be useful.
“For example, elderly people sometimes have physical therapy and it’s not always easy for them to travel,” Russ said. “So maybe we could put this system inside their home and this avatar can monitor their exercises and send the information back to the physical therapist who could see if they are doing everything that they are supposed to do.”
The first step was to determine if the Kinect’s “eye” is reliable enough for clinical care. For that, they would need a virtual trainer to lead people through exercise routines while the Kinect and a separate motion-capture camera collects data from their movements.
To build this, Russ found a partner in Capraro, a self-designed film and physics major through the Whittier Scholars Program, who could lend her coding and animation skills to the project. After intensive hours of research, troubleshooting, and guidance from Assistant Professor of Kinesiology Joshua Haworth, they developed a virtual physical therapy simulation and connected the program to both cameras.
“It was a good opportunity for me to learn and use my animation skills to enhance the project so that it looks more like a physical therapy clinic and not just robots doing some arm motions,” Capraro said. “It’s a little more comforting to the client.”
With the simulation and cameras in place, they then hooked up motion-capture sensors to 14 test participants and recorded their movements as the trainer led them through exercises in the Yao Yuan Sze Foundation Movement Laboratory.
After Russ analyzes the data from the cameras, they plan to partner with the mathematics department to take the project to the next level: giving the virtual trainer the artificial intelligence it needs to become effective.
“It’s amazing and it’s going to be revolutionary the more and more we develop it,” Capraro said.
As Russ sets her sights on graduate school, she hopes to continue investigating the benefits of combining physical therapy and the virtual world.
For her part, Capraro hopes to work at a visual effects production house to create the dazzling worlds within movies or video games.