Atlanta Jazz Robots

Shimon is a marimba player. Whether improvising on jazz standards or trading bars with a piano player, Shimon can switch styles and rhythms with ease. Like many musicians, Shimon grooves back and forth to the music, getting lost in the sounds of the instrument. Also, Shimon is a robot.

Watching video of Shimon using its robotic arms to hit the marimba keys, accompanied by a human on piano, is both eerie and magical, like an uncanny valley of sound. Music – particularly music that involves improvisation – often seems otherworldly, even spiritual, based in knowledge that is hard to write down, let alone program. But that’s exactly what Gil Weinberg, a professor of music technology at Georgia Tech who started its Robotic Musicianship program in 2009, is trying to accomplish. With Shimon, his team of students and faculty created a machine that can synchronize with human players, track beats and changes in key and improvise on chord changes, just like any trained jazz musician.

Instead of programming computers to produce sounds, like most electronic musicians, the Georgia Tech program has built robots that can play regular acoustic instruments. Weinberg hopes his robot musicians will connect with people in a deeper way. “The reason we go to shows, as opposed to just sitting in the living room listening to music from speakers, is because we want to see that interaction between musicians,” says Weinberg. “We want to see the guitar player jumping in the air, looking at the drum player to time the last chord of the song. That’s something we can do with robots.” Creating algorithms that emulate improvisational styles has not been easy, though. “Often times, when you try to code, the music that comes out isn’t as musical as you want, it doesn’t have the emotional connection,” Weinberg says. “The main challenge is to try to bring pieces of soul into robot music.”

The Georgia Tech program took the idea of human-robot collaboration to the next level when they teamed up with Jason Barnes, a drummer who lost his arm in an accident. Barnes already played with a prosthesis, but the Georgia Tech team built a custom robotic prosthesis that responds to electrical signals from the contractions in his residual muscles. After a year in which Georgia Tech built the arm and then worked with Barnes to calibrate the device, the drummer was able to try it out in 2014. Though he found the prosthesis awkward at first, Barnes was happy with his increased control over the drumstick. “It gave me back what I was missing, which was controlled strikes on a drum,” he says. Getting used to performing with a robot partner has still been a challenge. ”Some of it sounded cool, some of it sounded really bad,” he says. “It’s kind of a hit or miss surprise. I don’t ever know what it’s going to do, completely.”

Weinberg quickly realized that in addition to restoring Barnes’ natural abilities, the robotic arm could help him play the drums in a way unlike any human drummer. The prosthesis can hold two drumsticks, and when both are beating as quickly as possible they can get up to 40 beats per second. That’s at least double the highest recorded human speed, meaning Barnes can play polyrhythms that are physically impossible for any other drummer.

Inspired by Barnes, the robotics program is now developing a third arm that can be worn by anyone to give them the kind of superhuman abilities Barnes gained. This third arm will also connect to brain sensors that can predict what the human musician is thinking of doing next. Weinberg explains, “We were excited about the extra ability that Jason had and we said, ‘Why shouldn’t everyone have that?’” The third arm prototype is now complete and may be used for a performance at Moogfest, though Weinberg says it will be three to five years before it’s available to the public.

But it’s unclear if musicians will truly want a robotic partner accompanying them everywhere. Barnes told me he wasn’t too thrilled about his new prosthesis’ mind of its own. “Personally, I would like it with just one stick,” he said. “Then I would have complete control over it, which is what any drummer would want.” When I asked Barnes whether he thought the third arm could be a popular musical enhancement in the future, he was blunt. “Not really. Not at all, really,” he said.

“In the beginning, playing with the robots is kind of tricky,” Weinberg admits. “There’s a learning curve.” But he remains hopeful for the future of human-robot musical collaboration. “Some people ask me if I want a future where robots replace humans. That’s obviously not the future I’m looking for. It’s about using what robots are good at, which is processing and mechanical abilities, and bringing it together with what humans are good at, which is emotion and expression,” he says. “Together there will be a spark that can really create something new and exciting.”