Towards a Gesture Sound Cross Modal Analysis

From IMTR

Abstract

This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from sub jects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual sub jects.

In this page we present the data used in the article. We have used two sounds:

• an ocean wave
• solo flute playing a single note with strong timbre modulation (extract from Sequenza I for flute (1958), by Luciano Berio)

The following videos illustrate the gestures used for the analysis. The data has ben collected in May 2008 in the University of Music in Graz. We would like to acknowledge all the participants.

keywords: canonical correlation analysis; music; sounds; gesture; embodiment

Gesture performed on a wave sound

Gesture performed on a sound of flute