Gesture associated to a Sound
From IMTR
By the following studies, we aim at analyzing the gesture responses to some sound stimuli. Among musical gestures, there has been a great deal of recent interest in understanding and modeling instrumentalists' gestures. On the contrary, accommpanying gestures (or gesture responses to sound stimuli) have been underestimated and can highlight insightful strategies for the analysis of gesture-sound relationships.
Contents |
Explorative Study
Abstract
This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from sub jects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual sub jects.
In this page we present a video example illustrating a gesture performed while listening to the sound of a wave. Another example with a solo flute playing a single note with strong timbre modulation (extract from Sequenza I for flute (1958), by Luciano Berio) can be found here
The following videos illustrate the gestures used for the analysis. The data has ben collected in May 2008 in the University of Music in Graz. We would like to acknowledge all the participants.
keywords: canonical correlation analysis; music; sounds; gesture; embodiment
Example: the wave
Gesture Responses to Causal and Non-Causal sounds
Abstract
The presented work is part of a project aiming at modeling movement-sound relationships, with the end goal of proposing novel approaches for designing musical instruments and sounding objects. The experiment is based on sound stimuli corresponding to "causal" and "non-causal" sounds. It is divided into a performance phase and an interview. The experiment is designed to investigate possible correlation between the perception of the "causality" of environmental sounds and different gesture strategies for the sound embodiment. In analogy with the perception of the sounds' causality, we propose to distinguish gestures that "mimic" a sound's cause and gestures that "trace" a sound's morphology following temporal sound characteristics.
A Video Example
Further in the non-causal case: sound tracings
http://folk.uio.no/krisny/?page_id=177
Related Publications
2011
- K. Nymoen, B. Caramiaux, M. Kozak, J. Tørresen. "Analyzing sound tracings - amultimodal approach to music information retrieval" In ACM Multimedia – MIRUM 201, Scottsdale, Arizona, USA.
- B. Caramiaux, P. Susini, T. Bianco, F. Bevilacqua, O. Houix, N. Schnell, N. Misdariis. "Gestural Embodiment of Environmental Sounds: an Experimental Study" New Interfaces for Musical Expression (NIME 2011), Oslo, Norway. 2011
2010
- B. Caramiaux, F. Bevilacqua, N. Schnell. "Towards a Gesture-Sound Cross-Modal Analysis" In Embodied Communication and Human-Computer Interaction, volume 5934 of Lecture Notes in Computer Science, pages 158–170. Springer Berlin / Heidelberg, 2010