At CMU’s Panoptic Studio, 531 cameras capture the tiniest details of human movement
A two-story dome at Carnegie Mellon University outfitted with more than 500 cameras could help computers diagnose autism, fix a golfer’s swing and understand body language.
Hanbyul Joo and Ginés Hidalgo, two researchers in CMU’s Robotics Institute, have spent three years gathering data on human movement from inside the Panoptic Studio in the basement of Newell Simon Hall.
“When we are communicating, we are using our bodies as well to convey some information,” Joo said. “Each of our body gestures or facial expressions has some specific meaning, but that meaning is really hard to describe or define.”
That’s where the 531 cameras inside the Panoptic Studio come in. The cameras, a combination of high- and low-resolution rigs, capture human movement inside the dome. That movement is fed into a computer that catalogs and measures it, creating data about how we nod, smile, cross our arms, point our fingers and roll our eyes.
“Ultimately, we want to make machines to understand that kind of information by themselves,” Joo said.
Right now, the Panoptic Studio gives researchers a highly detailed, 3-D, 360-degree look at movement. The studio generates 600 gigabytes of data a minute when all 531 cameras are recording.
Athletes could have their swings, kicks, serves and running captured. Joo and Hidalgo have shot video of a dancer, a cellist, a guitar player and a drummer. Joo has watched his son grow up inside the Panoptic Studio, taking him in a couple of times a year to record how he plays, walks and interacts.
Joo and Hidalgo just released software that provides the same tracking and measuring for regular videos shot outside the studio.
RELATED: CMU researchers are teaching a computer to understand body language
This could mean detailed analysis of sports and insight into music, dance or how a child develops, Joo said. A doctor could look at footage of a child, run it through a computer program and determine, based on movement, whether the child is autistic.
And once Joo and Hidalgo collect enough data and develop the right algorithms and artificial intelligence, a computer could make the diagnosis, Joo said.
“To understand, we need to know what is normal interaction and what is not normal interaction,” Joo said.
Joo and Hidalgo’s research builds on decades of work by Takeo Kanade, a professor at CMU’s Robotics Institute. Kanade’s work into multi-camera imaging began more than 30 years ago with a handful of low-resolution cameras. In 2001, CBS debuted Kanade’s multi-camera technology during the Super Bowl. Viewers could see replays stitched together from 30 cameras. The image could rotate around the action.
Kanade in November was awarded the prestigious Kyoto Prize , Japan’s highest private honor for those who contribute to humankind’s scientific, cultural and spiritual development, for his work on computer vision, including pioneering and advancing the field of computer vision, critical to the development of self-driving cars.
Aaron Aupperlee is a Tribune-Review staff writer. Reach him at [email protected], 412-336-8448 or via Twitter @tinynotebook.