The Microsoft Kinect is the gadget du jour for music hacking.
We’ve previously featured several interesting Kinect music hacks on Synthtopia, including a Kinect air guitar, a Kinect virtual piano and a Kinect drawable synthesizer.
And these hacks seem to have wowed other sites and the mainstream tech press.
Nevertheless, we find ourselves wondering if the Microsoft Kinect is fundamentally useless as a musical tool.
Using a limited resolution sensor (640 x 480 pixel) with a relatively slow sampling rate (30 fps) means that the Kinect captures a fraction of the information that an older, simpler and cheaper “touchless” analog technology – the theremin – is capable of.
Here’s an example of a Kinect theremin:
While this is a clever hack – it also highlights the handicaps of limited resolution and a low sampling rate.
The limited resolution means that there’s a practical limit to the number of steps of control that the Kinect can afford. This shows up as stairstepping and “jitteriness” in the Kinect theremin demo.
More critical for musicians, though, is the Kinect’s latency. The frame rate alone of the Kinect is going to create 35-40 milliseconds of latency, In practice, though, the latency appears to be much higher, closer to 1/10th of a second.
Is The Microsoft Kinect Useless As A Musical Tool?
So is the Kinect useless as a platform for developing music apps?
The platform’s limitations mean that trying to imitate traditional musical instruments with the Kinect is probably going to be unproductive. The video demos for apps that take this approach seem to bear this out.
Nevertheless – there’s probably a lot of rich territory to mine with the Kinect, for developers that look beyond the obvious.
One area that may be rich for exploration is the idea of gestural music sequencing.
Morton Subotnick was exploring the idea of synthesizing music based on gestures 40 years ago. There are lots of more recent examples, including John Keston’s work with creating generative music based on gestures with his Gestural Music Sequencer.
Another area that offers potential is using the Kinect for virtual grid sequencing.
This video of NIN in action with Echoplex shows that latency is a problem that can be worked around, with very cool results.
What do you think of the potential of the Microsoft Kinect as a musical tool?
Do you think it’s useless – or only useful as a toy? Or do you think there’s more potential to the platform than is obvious? We’d love to hear your thoughts.
And if you know of interesting examples of the Kinect’s musical application, leave a link below. We’re interested in seeing more examples, whether they prove us right – or wrong.
This one's silly but pretty cool.
[youtube lqxUzhDIdqo http://www.youtube.com/watch?v=lqxUzhDIdqo youtube]
I dig it. I guess it's the same as most of these alternative methods of control, rather than them replacing instruments, which never works that well, having them as a way of providing a visual method of control over your sound has got to increase the impact of a live situation.
I don't see resolution as a real limiting factor, because I'd rather quantize or "autotune" the pitch anyway. My first pass at writing a Kinect Theremin in Processing used 24 notes vertically. Now I've scaled that back to two octaves of pentatonic scale, so I have room to be a little sloppy in my movement… (~48 vertical pixels per note).
I'm debating moving the pitch scale onto the X axis and playing it like JMJ's laser harp next.
Sample rate of 30 fps means at 120 bpm, u have a half note in 30 fps, quarter note in 15 frames, 8th note in 7.5, 16th note in 3.75, and 32nd note in 1.875 frames.
This is more than enough for 99% of musical uses.
The latency of 35 ms can definitely be an issue, however.
A tool is as good or useless as the person who's operating it.
true… so true
I would say the Kinect based solutions are as useless or useful as those based on ordinary off-the-shelf cameras or other forms of CCD sensors. Such tools are very useful in different professional domains (just take a look at medical imaging and navigation during complex operations) and that's why I have the feeling that basically they could be made to work flexibly and reliably for music making.
I've played with extracting motion information from videos a bit in the past, it's a fascinating field in its own right, and you can have a lot of fun (and a great learning experience in many regards) when building a video input program in something like OpenCV or Jitter (sorry, don't know any other ones from personal experience).
Whether or not you would really like to use video input for making all of your music instead of just as a gimmick or specialized tool for certain tasks it does well is as much a question as whether you need or would like to use the penultimate gadget of our time, the touchscreen. Both can be stunning in the hands of people who know what they are doing and utterly frustrating when someone who thinks they are easy and fun to use all the time tries to do something meaningful (i.e. what he or she had in his mind prior to thinking about the instrument at hand) with them. In this regard the comparison with the Theremin reveals more commonalities than it does differences.
It's useless to me because to use a kinect would also require bringing an xbox and tv screen anywhere you went. Furthermore, you would need appropriate space and lighting conditions for the kinect to function. The weight, unreliability and high potential for damage of that whole device chain is like going back to the early 90s to gig. On the other hand, any number of mobile devices offer far more options, benefits and performance, usually at a much lower cost.
Also, the software development and publications requirements from Microsoft ensure that music on the kinect will never be more than a passing curiosity. Move on to the next topic, everyone.
But there are some instruments that no one can make sound good…..
it's an avenue to explore that is intriguing. it has potential i think, but it's crude in its current state. if it were more interactive between multiple players I think that would be interesting. Also, the control of dynamics and the development of instruments that require a certain degree of technique to master would help Kinect and its affiliated programs earn some needed credibility and staying power in this world of quick apps with little usability. Without development and creativity however, Kinect would take its place amongst the graveyard of instruments that have a 'karaoke mentality'….the sense that it's just for amusement with little or no lasting meaning.