This video captures a performance of Tokyo Lick, by Jeffrey Stolet, using custom software and infrared sensors. The system converts his waving of the hands into complex piano music.
Stolet describes his system as a “new paradigm for virtuoso music performance”.
Details below.
Simple Input, Complex Output: Performance and Data mapping in Tokyo Lick.
Challenges regarding the conceptual design and implementation of human / music instrument interfaces have a rich and nuanced history. Generally, if a musical instrument has thrived it has been due to the fact that the particular instrument could provide the desired musical outcome. Traditional instruments typically display a simple one-to-one relationship in terms of input and output (e.g., one piano key is depressed, one note is sounded). Current technologies release us from the shackles of such one-to-one input-output models and permit to the creation of new types of musical generation. At the University of Oregon we have been involved with projects where musical robots perform music, where eye movement data control sound and video, and where infrared sensing devices control sonic and video events.
In his program, Mr. Stolet will focus on the technology and the human-performance elements in Tokyo Lick, his composition for infrared sensors, custom interactive software, and MIDI piano. He performs Tokyo Lick by moving his hands through two invisible infrared spheres and directing the data derived from those motions to algorithms residing in customized interactive software created in the Max multimedia programming environment.
Tokyo Lick contains no sequences or pre-recorded material. Mr. Stolet will perform every note in real-time. Using a technology he refers to as “algorithm flipping,” he can rapidly change the specific algorithm or algorithms governing the response to the incoming MIDI control data. He actuates the algorithmic changes through pre-composed schedules, musical contexts, or through explicit intervention. Taken together, these techniques provide a conceptual framework for practical input/output mapping (action ? specified outcome) and for control and performance flexibility, while offering a truly new paradigm for virtuoso music performance.
via deviantsynth
I think the key phrase is "pre-composed schedules". I've experimented with using the built-in camera in my MBP to use motion detection (frame deltas) as controller data, you don't get very fine control but with infrared it is probably much better.
So with scheduled algorithm switching and proper motion detection data to pump into the algorithms, this kind of performance does not appear too challenging.
This is a response to the comments of nuclearsound (above).
nuclearsound is incorrect. No faking … only practice.
Using a Yamaha Disklavier (or a surrogate synthesis module), two infrared sensors, and
two MIDI pedal controllers, I perform the piece by moving my hands through two
invisible infrared spheres and directing the data derived from those motions to algorithms
residing in customized interactive software created in the Max programming
environment.
There are no sequences or pre-recorded aspects in Tokyo Lick. Every note is performed in real-time. I can perform the piece virtually the same each time or quite differently each time.
To get the musical variety I remap those measurements of distance to pitch and to musical dynamics shuffling different parts of the piano keyboard under my hands as I perform. [Each time I move a different part of the piano keyboard under my hands the distance measurements are mapped to different parts of the keyboard.]
Tokyo Lick is enormously difficult piece to perform. If I do not practice it for several months, it takes me more than a week of rehearsing to again work the piece beyond the first minute.
Thanks for listening.
Jeffrey – the challenge here is that people don't understand the mapping between your physical gestures and what they are hearing, like the do when they see someone play a traditional instrument.
That's a fundamental challenge for laptop electronic music performance.
If you're not waving your hands around, people think you're checking your email. If you are gesturing wildly, people have trouble making the connection.
Jeffrey – the challenge here is that people don't understand the mapping between your physical gestures and what they are hearing, like they do when they see someone play a traditional instrument.
That's a fundamental challenge for laptop electronic music performance.
If you're not waving your hands around, people think you're checking your email. If you are gesturing wildly, people have trouble making the connection.
What aer the algorithms used? Do you use some special kind of algorithms, Brownian movements, fractals, or are they self-invented algorithms?
I think what makes this more difficult to appreciate is:
1. Some of the hand motions look really broad and imprecise while others look subtle, yet the music doesn't correspond accordingly.
2. It seems like you could achieve similar results with simple IR beams triggering a few "sample and hold" modulators.
I'm not calling bullshit on this, just explaining why the video may not convey how complex the actual performance is.