Workshop Engineering and Music

"Human Supervision and Control in Engineering and Music"

Teresa Marrin Nakra

Translating Conductor’s Gestures to Sound

Abstract

There are many ways in which engineering can be used to enhance and extend the practice and experience of classical music. The author’s expertise in conducting was used as a point of departure and extension into electrical engineering, whereby a series of studies and electronic performances of conducting were undertaken.

Introduction

Ten years ago, I was a college student studying classical music, focusing on conducting, composition, and theory. I conducted several opera productions and student orchestras, and had a great love for new music. I was excited about the advances that were being made in electronic music, but didn’t know how to make sense or use of them. Through a coincidence I learned about the work of Tod Machover at MIT, who at the time was wiring up a cello for a new piece to be performed by Yo Yo Ma; in the process he was measuring the characteristics of Yo Yo Ma’s playing style and discovering unique aspects of his technique. This sparked my imagination and got me thinking about other possibilities for adding technology to classical music. For six intervening years I worked at MIT to build systems to perform, study, and analyze music, particularly from the perspective of orchestral conducting.

Ten years later, I am here to tell you that there are a tremendous number of ways in which engineering (particularly the disciplines of electrical engineering) can be used to enhance and extend our practice and experience of classical music. This area remains rich with possibilities, as those of us who work in it have only just begun to scratch the surface. For example, by continuing to develop better methods to sense and analyze musical performances, we can help students by giving them quantitative tools with which to evaluate their performances. Also, by sensitively and carefully integrating amplified, synthesized, and sampled sound into live performances we can increase the overall palette of available sound. We can also add special effects and visuals to musical performances, where the behavior of the visual elements is driven by the shape of the musical performance. And by continuing to improve upon gestural control systems, we can ensure that performers will not have to change their established techniques -- just perform naturally and let the computer systems follow them. We can also start to truly study and explain in-depth many elusive questions, such as how music contagiously conveys emotion and expression from performer to audience.

In addition to performance enhancements, there are other good reasons to add new technology to concerts. Audience interest for classical music (at least in the US) has decreased considerably
during the past thirty years. This has partially been blamed on the dissonant and complex music of the 20th century that has alienated audiences. Also because of negative audience reactions to modern music, orchestra repertoire has stagnated over the 20th century, and we all have become comfortable with standard 18th- and 19th century compositions that are repeated many times over. Meanwhile, during the past ten years there has been a tremendous surge in the public’s use of computers and new technologies. Therefore, it seems that if classical musicians can find ways to sensitively and intelligently incorporate new technologies into their work, they will capitalize on the enthusiasm of new generations of technology-savvy audiences.

Background

There are many who have worked in this area between classical music and engineering, and their accomplishments deserve discussion. Max Mathews, the legendary Bell Labs researcher and founder of CCRMA at Stanford, began experiments with his Radio Baton system during the 1960s.[Mathews 1990, Mathews 1991] He continues this work today and is constantly improving his gesture-recognition and synthesis algorithms. Also, in the early 1970s Gerhardt Harrer embarked on a physiological study of the great Berlin Philharmonic conductor, Herbert von Karajan.[Harrer, 1975] One of his most remarkable results showed that Karajan’s heart rate was higher when conducting Beethoven than when flying his airplane under dangerous conditions. Professor Tod Machover, my doctoral advisor at the MIT Media Lab, has also contributed significantly to the field with his Hyperinstruments research projects in a variety of performance areas.[Machover and Chung 1989, Machover 1992]

My own first attempt to use electronics to help me “conduct” music was a device called the Digital Baton.[Marrin 1996, Marrin 1997, Marrin and Paradiso 1997, Marrin et al 1999] Designed by a team at the MIT Media Lab, the baton had 11 degrees of freedom (3 axes of acceleration, 3 of position, and 5 points of pressure sensitivity). I designed it to resemble a traditional conducting baton with an enlarged handle, but inevitably it was too large and heavy to be comfortably used by a conductor. Its wire also made it easy to trip on, which was not ideal for performances. However, most importantly, the Digital Baton taught us that we knew very little about the actual mechanics of conducting; how to determine a downbeat from a beat-2 or beat-3; how to demonstrate piano versus forte, and how to cue various instruments. Any musical mappings that we made were entirely based on guesswork and trial-and-error. This was a fatal flaw that led me to my second project in this area.

In the fall of 1997 I began the Conductor’s Jacket project.[Marrin Nakra 2000, Marrin Nakra 1999, Marrin and Picard 1998, Marrin and Picard 1998] The idea behind the jacket design was to sense as many significant aspects of conducting as possible through sensors embedded in the clothing of the conductor. The wearable form factor was ultimately successful, because it didn’t interfere with the natural movements and expression of the individual, and we were able to take measurements at the surface of the skin that indicated internal states in addition to movement parameters. I undertook numerous data collection experiments with the jacket and continue to actively work with it as a platform for research, education, and performance.

Projects in Gestural Control of Sound

Recent and ongoing performance projects with the Conductor’s Jacket include a performance with the Boston Pops Orchestra, early conducting synthesis programs, collaborations with Manfred Clynes and his software system called “Superconductor,” and a commissioned “Concerto for Conductor” by composer John Oswald. In addition, for the past 1½ years I have developed an education project in collaboration with Arizona State University. Called the Digital Conducting Laboratory, this project is an extension of the Conductor’s Jacket that uses EMG sensors and a set of etudes to systematically review basic conducting gestures and help teach students how to conduct. Using a digital conducting feedback system, the lab simulates many of the behaviors of a live orchestra rehearsal setting. An interactive program recreates several fundamental ensemble-conductor interactions, including tempo, articulation, and dynamic line, and allows conductors to review their performances via sound files, video playback, and analysis of muscle-tension profiles.

Future Directions

Future work on the Conductor’s Jacket system will include extensions such as the integration of visual systems (including animations, lighting, and digital video), a wireless transmitter (for increased mobility for the performer), and the addition of more sensors to capture different gestural features. Also, now that the jacket method has been shown to work for one group of musicians, it makes sense to investigate whether or not it will work for others, such as singers and instrumentalists, dancers, acrobats, and ultimately the general public. Much more research and data collection should be done, including a follow-up to the research in my doctoral thesis.

Some of these ideas have already been tried out, to a limited extent. In the spring of 2001, I wired up violinist Joanna Kurkowicz and built a MIDI lighting system (designed by Herrick Goldman) to reflect the character of her movements and performance. Collaborations with real-time digital video artist Walter Wright have been ongoing for almost two years. Finally, my organization is planning to launch a new music group, to be called the Immersion Music Ensemble, in the fall of 2002, to specifically present new kinds of technology-enhanced music.

Conclusions

In this elusive space where classical music and engineering meet, there is fertile ground and much work to be done. Classical music is a historically significant and information-rich tradition, but lacks appeal and relevance in modern society. By sensitively developing tools that enhance the performance and experience of classical music, we are perhaps revitalizing a tradition that deserves to be continued and extended.

Music is an extremely complex and subtle art; it requires advanced engineering methods to capture and represent it in enough detail and sophistication. Figuring out how to quantify and reflect what musicians do when they express themselves in their art form is a very interesting and fruitful area. I would encourage all of the musicians, conductors, and theorists who have not yet tried to use engineering techniques to make the leap as I did nearly ten years ago. You will certainly be delighted with the results.

References

Harrer, G. (1975). Grundlagen der Musiktherapie und Musikpsychologie. Stuttgart, Gustav Fischer Verlag.

Machover, T. (1992). Hyperinstruments: A Progress Report, 1987-1991. Cambridge, M.I.T. Media Laboratory.

Machover, T. and J. Chung. (1989). Hyperinstruments: Musically Intelligent and Interactive Performance and Creativity Systems. International Computer Music Conference, page 186-190.

Marrin Nakra, T. (2000). Inside the Conductor's Jacket: Analysis, Interpretation and Musical Synthesis of Expressive Gesture. Ph.D. Thesis, Media Laboratory. Cambridge, MA, Massachusetts Institute of Technology. http://immersionmusic.org/HTMLThesis/Dissertation.htm

Marrin Nakra, T. (1999). Searching for Meaning in Gestural Data: Interpretive Feature Extraction and Signal Processing for Affective and Expressive Content. Trends in Gestural Control of
Music. M. Wanderley, ed. Paris, IRCAM.

Marrin, T. (1996). Toward an Understanding of Musical Gesture: Mapping Expressive Intention with the Digital Baton. M.S. Thesis, Media Laboratory. Cambridge, MA, Massachusetts
Institute of Technology.

Marrin, T. (1997). Possibilities for the Digital Baton as a General-Purpose Gestural Interface. CHI '97 Conference on Human Factors in Computing Systems, pages 311-312.

Marrin, T., Joseph Paradiso, et al. (1999). Apparatus for Controlling Continuous Behavior Through Hand and Arm Gestures. United States Patent no. 5,875,257, issued February 23, 1999.

Marrin, T. and J. Paradiso. (1997). The Digital Baton: a Versatile Performance Instrument. International Computer Music Conference, Thessaloniki, Greece, pages 313-316.

Marrin, T. and R. Picard. (1998). Analysis of Affective Musical Expression with the Conductor's Jacket. XII Colloquium for Musical Informatics, Gorizia, Italy, pages 61-64.

Marrin, T. and R. Picard. (1998). The Conductor's Jacket: a Device for Recording Expressive Musical Gestures. International Computer Music Conference, Ann Arbor, MI, pages 215-219.

Mathews, M. V. (1990). Three dimensional baton and gesture sensor. U.S. patent # 4,980,519, December 25, 1990.

Mathews, M. V. (1991). The Conductor Program and Mechanical Baton. Current Directions in Computer Music Research. M.V. Mathews and J. R. Pierce, eds. Cambridge, MIT Press.