Workshop Engineering and Music

"Human Supervision and Control in Engineering and Music"

Thomas Hermann

Sound and Meaning in Auditory Data Display

Abstract

This paper focusses on the connection between human listening and
datamining. The goal in the research field of datamining is to find
patterns, to detect hidden regularities in data. Often, high-dimensional
datasets are given which are not easily understood from pure inspection of
the table of numbers representing the data. There are two ways to solve
the datamining problem: one is to implement perceptional capabilities in
artificial systems - this is the approach of machine learning. The other
way is to make usage of the human brain which actually is the most
brilliant data mining system we know. In connection with our sensory
system we are able to recognize and distinguish patterns, and this
capability is usually exploited when data is presented in form of a
visualization. However, we also have extremely high-developed pattern
recognition capabilites in the auditory domain, and the field of
sonification addresses this modality by rendering auditory representation
for data for the joint purposes of deepening insight into given data and
facilitating the monitoring of complex processes.

An unanswered question is how high-dimensional data could or should sound.
This paper looks at the relation between sound and meaning in our real
world and transfers some findings onto the sonification domain. The result
is the technique of Model-Based Sonification, which allows the development
of sonifications that can easily be interpreted by the listener.

Auditory Perception and Environmental Listening

In evolution, the auditory senses developed because they provide us with
relevant information to survive. However, we use listening far beyond this
primary purpose by developing language and music. These three sorts of
auditory signals shall be distinguished by their different functions.

The elementary and oldest function of listening is the detection of sound.
Sounds in our environment provide us with awareness and they are able to
draw our attention to potentially dangerous events (e.g. approaching
enemies). Besides this, sound allows us to extract lots of information
about a 'sounding' object: its size, material, surface, tension and so on.
We are able to abstract from sounds to properties of the sounding object or
sound process because the connection between sound and an object is fixed,
given by the laws of physics. Because the coupling mechanism (physical
laws) were constant over a large time scale, evolution was able to develop
'hard-wired' mechanisms to interprete such signals and pull out relavant
information from the signal without the need for conscious processing.

Sound emerges as a consequence of excitation of physical objects, thus
objects/instruments in equilibrium are usually silent. Humans often excite
objects consciously and thus get sound as a feedback to their actions. So
they can relate the sound to their actions and learn about the world from
this interaction loop. E.g. pressing a button, we know from the sound
(besides haptical feedback) if our action succeeded. While these
observations seem self-evident, they are often ignored when considering
sonification and techniques to access data by acoustic
representations. Using this relation between sound and meaning, an
alternative to Parameter Mapping, the prevailing sonification technique, is
developed in the next section.

Let us now focus on the relation between sound and meaning within other
types of sound signals. In language, spoken words receive their meaning
within a cultural context and the association is learned by each child.
The relation between the sound of the word 'table' and the meaning of this
word must be learned and is somehow arbitrary. Obviously, humans have also
excellent capabilities in learning and accessing the meaning of learned
auditory patterns. While the information within environmental sounds is
analog, language emphasizes the communication of symbolic information or
abstract content. In sonification, verbal messages are suited to label
categorial data or provide symbolic labels within an analog auditory data
display.

Musical Information lies in between these two sound types. Controlling
sounding objects by human supervisors leads to sound that both gives
information about the instrument and the performer. Harmonic relations find
an analog in physical laws (Fourier decomposition of periodic signals)
while melodic and rhythmic structures are related to prosodic patterns in
language and narrative.

Data Sonification

High-dimensional data is given by vectors of numbers. The question is how
to generate meaningful sounds from given data. The traditional approach is
to synthesize a sonification by superimposing sonic events whose attributes
(onset, frequency, duration, amplitude, etc) are mapped by numeric values
of the given data vector. This technique is known as parameter mapping. The
main disadvantage is, that there is no canonical way to specify the mapping
and that the mapping must be known prior to interpreting the
sonification. For high-dimensional data, a mapping might be long and
complex. There are further problems with interfering perceptional
dimensions.

Model-Based Sonification

Model-Based Sonification solves some of the problems mentioned above by
providing a structured method to link data to a sounding object. Different
from parameter mapping, data is taken to parameterize a sound model, which
rests in a state of equilibrium without excitation. Dependent upon the
model, dynamical laws are introduced which cover the temporal evolution of
the model, giving rise to acoustic signals which represent the
sonification.

The main advantages of this model-based approach is, that the sound and its
meaning with respect to the data are connected in the same way as in the
real world. Thus, intuitive metaphors can be applied to interact with the
model. Think of a model where the local tension of a membrane surface is
parameterized by data. It can be struck, plucked, rubbed, etc. to make it
produce sound. Information about the data is given in the control loop
between human excitation and system reaction. As a model definition is not
bound to specific data, humans get the chance to become familiar with the
sound space of a model. Furthermore, the model can be applied to arbitrary
data. If dynamical laws are applied which resemble physical laws that
govern sounding objects, the sonifications are likely to be sounds within
the soundspace we are familiar with from our listening experience.

Sonification models emphasize acoustic signals as a feedback to human
actions. This offers new perspectives for human supervision of complex data
and control of data manipulations. The investigation of sonification
models and their utility for exploration of high-dimensional data is the
topic of current ongoing research.