Workshop Engineering and Music

"Human Supervision and Control in Engineering and Music"

Jens Herder

Applications of Spatial Auditory Displays in the Context of
Art and Music

Abstract

Auditory displays with the ability to place virtual sound sources into the space under realtime conditions enable advanced applications for art and music. The
listener can be immersed on a high level while inter-acting and even participating in the experience. We review some of those applications and discuss the re-quired technology. Keywords: spatial auditory displays, interaction, spatial events, soundscape

1 Introduction

The space and the audience have an influence on the creation of music. Composers have written pieces, targeting a specific performance situation. While recordings allow a larger audience to appreciate certain music.
Even with the latest technology, recordings are not comparable to a live concert. A recording cannot reproduce the interaction between audience, space, and performers. Auditory displays with the ability to place virtual sound sources into the space under realtime conditions may allow interaction between audience and space. We give some examples of spatially arranged music elements in Section 2. In Section 3, we look at new applications in the field of art using spatial auditory display. Finally, we review virtual concerts applications in Section 4.

2 Using spatial arranged musical
elements in composition

Composers take advantage of spatial imagery and try to immerse the audience. In the following paragraphs, we give some well-known examples.

In 1829-1830, Hector Berlioz wrote the Symphonie Fantastique [Ber30] in which spatial arrangements are an important part of the music. In the “Scene in the country” two shepherds piping in the distance to each other. Here players of the pipes are separated from the main orchestra, creating a wonderful spacious imagery [NM80].

Karlheinz Stockhausen [Sto56] used in his “Gesang der J¨ unglinge” a multi channel recording and playback system, creating spatial movements in his composition.

Andrew Lloyd Webber’s musical cats [Web81] plays at special arranged theaters (e.g., Hamburg) in which the stage is around the audience and performers move freely close and behind the listener.

While a listener has individual experiences, his in-teraction or participating in the performance is limited. Art installations, which are reviewed in the next section go beyond those restrictions.

A much better review of the history of music space and reverberation in general is given by Barry Blesser [Ble01].

3 Art installations

Virtual environments are influencing real art [Dol97] [Bro97] and new technology is used for various instal-lations.

Feedback and complete isolation are explored in the installation of Seiko Mikami [Mik97]. The heartbeat of the visitor is spatially auralized and visualized. It takes place in an anechoic chamber providing total isolation.
The sound wanders around the listener and depends on his reaction.
Distance plays an important role in the installation of Akitsugu Maebayashi [Mae97].

Distance between people is visualized and auralized via a personal spatial auditory display. Participants of the installation wander around in a virtual (controlled) environment, consisting of a HMD, location tracking, and headphone.

Concrete location of sound is not always desired. In the interactive installation of Yunchul Kim [Yun01] the sound moves and is influenced by the visitors
movements (aka. position), but the sound seems not come from a specific location instead an omnipresent floating soundscape is created. A video-based person tracking controls a realtime composition using self-organizing maps.

A helical keyboard [HC96] can visualize and auralize the cyclical nature of octaves and helical structure of a scale, as described by [She82]. This system takes as input a Midi stream, either preproduced as Midi file or live performed. The notes are visualized in space by movement and color. Each note is a distinguished sound source, allowing to differentiate a note also spatially.
A listener can fork his listening presence in the virtual environment as shown in Figure 1. This al-lows to follow harmony and melody while keeping the
source direction differentiable.

4 Spatial emancipation of the audience

With spatial auditory displays allowing realtime com-position of the experience, the audience can emancipate itself spatially from the prefabricated (spatially
composed) content.

Virtual concerts are possible with the multiple audio windows [CK95] [Coh94], in which the listener can rearrange the (virtual) sound sources and sinks, creating an imagery of his own choice.

Figure 1: Exocentric visual perspective of forked presence
via multiple sinks in the Helical Keyboard

Virtual concerts are also presented in virtual environments [CK99] allowing also the listener to take active part in the concert by substituting one source.
Creating content for such systems requires that all voices be recorded separated and anechoic. In [Kau99] a violin concert was recorded in an anechoic chamber. Each player was recorded separately while listening via headphone to the other voices for synchronization and harmonic play.

With numerous tracks available on DVD and tagging of voices, and consumer spatial auditory display becoming available, vitual concerts can become another
common listening experience [Her00].

References

[Ber30] Hector Berlioz, Symphonique fantastique, 1830.

[Ble01] Barry Blesser, An interdisciplinary synthesis of reverberation viewpoints, in press (2001).

[Bro97] Sheldon Brown, Real Art and Virtual Reality, Computer Graphics 31 (1997), no. 4, 36–39, FO-CUS: The State of Fine Art.

[CK95] Michael Cohen and Nobuo Koizumi, Audio Windows for Virtual Concerts II: Sonic Cubism, Video Proc. ICAT/VRST: Int. Conf. Artificial
Reality and Tele-Existence/Conf. on Virtual Reality Software and Technology (Makuhari, Chiba; Japan) (Susumu Tachi, ed.), ACM-SIGCHI (TBD), SICE (Society of Instrument and Control Engineers), JTTAS (Japan Technology Transfer Association), and NIKKEI (Nihon Keizai Shim-bun, Inc.), November 1995, p. 254.

[CK99] Hartmut Chodura and Arnold Kaup, A Virtual Environment for Interactive Music Reproduc-tion, Proceedings of IFIP TC/WG5.10 and CSI
International Conference on Visual Computing 1999 (S.P. Mudur, D. Shikhare, J.E. Encar-nacao, and J. Rossignac, eds.), International Federation for Information Processing (IFIP), February 1999,
http://www.igd.fhg.de/igd-a9/research/audio/index.html ,pp. 95–100.

[Coh94] Michael Cohen, Conferences, concerts, and cocktail parties: Besides immersion, JMACS: Proc. Japan Music And Computer Science Society Meeting (Musashino, Tokyo), February 1994, pp. 17–26.

[Dol97] Margaret Dolinsky, Creating Art Through Virtual Environments, Computer Graphics 31 (1997), no. 4, 34–35, FOCUS: The State of Fine Art.
[HC96] Jens Herder and Michael Cohen, Design of a Helical Keyboard, ICAD’96 — Int. Conf. on Auditory Display (Palo Alto, CA; USA) (Steven P.
Frysinger and Gregory Kramer, eds.), November 1996.

[Her00] Jens Herder, Interactive Sound Spatialization - a Primer, MM News, University of Aizu Multimedia Center 8 (2000), 8–12, (Japanese).

[Kau99] Arnold Kaup, 3D Musikproduktion für Virtual Reality Anwendungen, Master’s thesis, Fachhochschule / Robert-Schumann Hochschule, Düsseldorf, 1999.

[Mae97] Akitsugu Maebayashi, Audible distance, Installation at the NTT InterCommunication Center (ICC), Hatsudai, Tokyo, 1997,
http://www.ntticc.or.jp/Collection/Icc/AUDIBLE/index.html.

[Mik97] Seiko Mikami, World, membrane and the dismembered body, Installation at the NTT InterCommunication Center (ICC), Hatsudai, Tokyo, 1997, http://www.ntticc.or.jp/Collection/Icc/World/index.html.

[NM80] New York Philharmonic Orchestra and Zubin Mehta, Hector Berlioz: Symphonie fantastique, op. 14, CD, The Decca Record Co. LTD., 1980,
MCPS 400046-2.

[She82] Roger N. Shepard, Structural Representations of Musical Pitch, The Psychology of Music (Diana Deutsch, ed.), Academic Press, December 1982,
ISBN 0-122-13560-1, pp. 343–390.

[Sto56] Karlheinz Stockhausen, Gesang der Jünglinge (song of the youths), 1955/56, http://www.stockhausen.org/ tape loops.html.

[Web81] Andrew Lloyd Webbers, Cats, Musical, 1981,
http://www.stella.de/cats/de/html/.

[Yun01] Kim Yunchul, Implex, Installation at Altitude 2001, Academy of Media Arts, Cologne, Jul 2001.