Technological and Computer-based Projects

AirSculpt: a handworn multiparameter proximity- and gesture-based control system for new physical and modeled instruments
Proposed project

Background: I am interested in creating highly sophisticated control systems that access many parameters of artistic creation at once: that allow for true artistry and real-time response in the case of professionals playing newly-invented instruments, or seeking to "wrap" one art form around another; and that bring intuitive control over an experience or creation to the layperson.

To take a simple example for the layperson: imagine music writing and orchestration software that not only lets you move pitches from high to low, but as you move them intelligently reassigns them among the orchestra's instruments; not just lets you make notes louder or softer, but assigns them to a solo, single section, choir or doubling, or tutti as it also changes something simple, like velocity; that keeps arrangements clear by intelligently dropping, say, passing and neighbor tones if a piccolo part is moved to the basses, or gives the option to play the melody at a slower speed, adapting the ostinato at tempo to work with the slower melody.

AirSculpt is a proposed hand-based device to serve as a optimized control interface for non-traditional new instruments such as The Chandelier, for which keyboard MIDI control does not approach being able to control the full range of possible expression. Even an instrument like guitar is poorly controlled by MIDI— a guitarist has the choice to play most notes on the instrument across several strings, and that choice radically changes the timbre of the instrument. A single line can be played by shifting the hand to keep on one string, or by crossing over strings to create a natural brightening or darkening of the sound. While this is a drastic tone change, even professional musicians often aren't consciously aware of it because it is so well-integrated into the instrument's sound, so consistent between performances, and so well-modulated by the performer that its only real effect is to increase the expressivity of the music being made. The goal of AirSculpt is to bring that expressivity to new instruments.

While this is an ambitious project that will doubtless involve phases of experimentation, testing, and revision, my first instinct is to control five aspects of music with a single hand: pitch, attack/velocity, aftertouch, brightness of timbre, vibrato/modulation, and patch changes. My performer's intuition leads me to keep pitch tied to distance from an antenna/object, as the largest distances moved correspond to pitch in all string instruments, piano, pitched percussion and, although the distances are smaller, this holds in wind instruments as well. It is only in non-pitched percussion that a player might move further to increase velocity than they move to change drums.

MagicMouse, a 2007 Popular Science Invention of the Year, is an elegant implementation of technology and ideas somewhat similar to what might be used in AirSculpt. Most interestingly, the project uses ultrasonic transmission from the hand-worn unit, computing differences in arrival time to several recievers. This data stream allows reconstruction of the user's hand position and movements in 3D space. The WPI team developed this as a replacement to the mouse for GUI interaction; I see this as a potential lightweight solution for AirSculpt's position indicator. (XYZ accelerometers will take care of tilt, pitch, and yaw, regardless of the hand's absolute position.)

    

MagicMouse photos used by permission from WPI. Thanks, WPI!


Control Strategy

In a traditional Theremin, note attacks and volume are controlled by a separate hand, with distance from a separate loop-shaped antenna corresponding to volume, and the player touching the loop to turn the note off completely. This has the disadvantage of making a simple sharp attack a bit of a virtuoso move, requiring the player to remove their hand from the loop very quickly while maintaining proper distance with the other hand for their note to sound in tune. My plan is to leave the index finger free to serve as a pitch-distance signaler, and use the thumb's sensor to trigger notes, almost as though plucking an invisible harp string in the air. The player can then "pluck" a note and release the thumb to invoke a standard decay envelope, or hold the thumb to the place it hits the second and third fingers to sustain the note, possibly altering other parameters at the same time. This gesture has the twin advantages of being musically intuitive and also allowing all other motions of the hand involved in controlling the AirSculpt except patch changes.

The third, or middle, finger can be bent to control expression/modulation, either continuously or as an on/off control, depending on the sound being produced. In controlling synth sound this would be a fairly straightforward mapping of a control onto a parameter. However, in controlling a new physical instrument such as The Chandelier, or an emulation of an instrument such as the guitar, this finger would likely be the player's first stop in controlling, of various possible strings capable of producing a given pitch, which to use. I say here "controlling" rather than "choosing" because unlike playing a guitar, the player should not have to pick a specific string (although as virtuosi of these new instruments emerge, they may desire/develop such control) but merely squeeze tighter for a brighter sound, or relax the finger outward for a darker one . Software, then, will control which string or string will give the desired sound, or even the best way to transition from the sound currently being made to the one that has just been called upon.

In this sketch of AirSculpt's possible configuration, I allow each of the three most dexterous fingers to control a parameter. I expect this control to be intuitive and as easily mastered as any new instrument given to a multi-instrumentalist used to remapping complex training onto new musical responses. That said, I suspect that players will be glad to not have major expressive functions mapped onto the fourth and fifth fingers. These will be only used to set patch changes, allowing a performer to alter the sound without bending down to touch a dial, or even while onstage performing in another capacity. I use two fingers so that each may enter a digit using repeated taps: to call up Program #42, one taps the fourth finger four times and the fifth finger twice. Because performance settings are likely to require less than 99 patches without some sort of set change or break, setting up patches to contain low numerals, say, no numerals higher than four or five would ease the burden of rapid weak-finger tapping on the performer.

Development, Refinement, Deployment

First steps with this interface would by synth-based, aimed at simply producing proof of concept and reliable triggers, workable by multiple musicians, whose functions are calibrated for the expectation of trained performers. Promising work has been done in this area and this phase should be readily achievable in a short timespan.

The next step would be mapping the modalities of this new controller onto the personality of the new instrument it is to control. This largely subjective but also highly technical undertaking falls into the category of things that are unlikely to fail outright, but will require intense dedication to raise to the level of an art; to enable true artistic control of a new instrument. That said; my experience as a multi-instrumentalist and designer of new creative technologies makes this a task toward which I am excited to work.