Project grant awarded to create virtual video orchestral conducting

Instead of conducting an orchestra from a concert hall, IU music informatics professor Chris Raphael will bring the orchestra to a computer screen.

Raphael received a three-year, $450,000 grant from the National Science Foundation for his proposal, “Real-Time Planning of a Conductable Orchestra,” said Lisa Herrmann, manager of communications for the School of Informatics.

The culmination of the project will be a computer program that is able to understand the gestural language of musical conducting through video, according to a press release. The program will create audio of an ensemble being led by the live conductor, Herrmann said.

“The idea is we want to make a tool that a person can use to imagine they are conducting an orchestra,” Raphael said. “Conducting has pretty well-defined gestural language – symbols they use for making cutoffs and making dynamic changes. We can make a virtual orchestra that will follow a conductor, so you can communicate the interpretive content, the expression of the music and learn to follow and understand gesture.”

John Poole, a senior lecturer specializing in choral conducting in the Jacobs School of Music, will work on the project with Raphael along with the help of a Ph.D. candidate from the School of Informatics. 

“The idea of dealing with real-time performance systems is something I’m quite familiar with,” Raphael said. “But after talking with John Poole, the idea for the project really sprung out from our interests at hand.”

Raphael’s proposal received the grant because of the project’s broader impacts – it will be able to run on generic computer hardware.

“This work will have lasting impact on conducting pedagogy by providing a tireless and responsive laboratory for musical experimentation,” according to the National Science Foundation award abstract. “Perhaps most importantly for society at large, a successful conducting system would bring the pleasure of music-making to a broad and international collection of users who might otherwise have little or no experience creating music.”

Part of the grant includes funds to purchase a reproducing Bosendorfer piano that can collect information about the key and pedal movements of the pianist, Herrmann said. Its abilities will allow researchers to analyze and reproduce a performance.

“The piano is especially terrific, and it’s great because the music school is interested in it, too,” Raphael said. “It provides a natural bridge between what we do and what’s done in the music school. It’s not the start of collaboration with the music school, but it will move us farther in that direction.”