Talía Baños Sánchez from Spain /MA. Music Pedagogy candidate
Marcel Vesga Gerardino from Colombia /MA. Digital Media candidate

Hochschule Für Künste, 2012
Bremen – Germany

This Masters Project Abstract may be cited as follows:

AVAMTI: Visually Assisted Musical Training Interface.
(Work in progress)

Project Abstract (Summary):

Practically unexplored and briefly applied yet, the inclusion of Digital Media within the instrumental pedagogy is creating a big potential to approach musical learning in more engaging and creative ways.

Our aim is to design an experience, which will allow beginners to train musical skills from an early stage in a meaningful and playful way exploring the correlations between music and sound, through natural learning processes of sensorial exploration generating an interrelation between the body motion and sound visualisation.

How does this process work?

We started to analyse the sound of the open violin strings: G, D, A and E. frequencies faraway from each other, to pick them up easier,we analysed consecutive notes a little bit faster.

We have a feature extractor analyser which captures sound through a standard laptop microphone. We focused on the “Raw Midi pitch” which basically tell us at which frequency is the violin vibrating. This midi-data is been translated into OSC (Open Sound Control) messages which can be later used to control visual environments in order to produce sound visualisation.

We approached  movement in a similar fashion with the Kinect. Different features from the performers’ motion are captured such as joint’ positions and speeds from both arms (hand, elbow and shoulder) being translated into OSC messages a well. All this information can now be used and read by sound synthesis or animation software such as “Processing” or “Quartz composer”.

Concept:

Despite music is not a language, as it has no grammar, is although learned as one, by listening and repetition; having a closer look at contemporary music teaching methods, we noticed these rely not only on the musical hearing experience, but are usually combined with other activities or mediums which will complement the musical learning experience.
Through a series of basic exercises, the students can train hearing and motor skills required to perform a musical instrument aided by a visual interface, which could be used in the classroom as well as in daily practice.
The next steps include translating information into visual representations from both, movements and sound.

How it all started…

This project idea started from a shared interest in having a collaborative experience between both our disciplines; we believe and would aim to take on an exploration in how an alternative method involving multimedia will prove to become effective and enrich musical learning experiences, thus reinforcing the interdisciplinary collaboration between the fields of the arts.

As our research started we were looking for connexions between Design and Musical Learning, the Visual and the Auditive. An obvious approach came to appear as we stumble upon the paper “Synaesthetics – a basis for meaningful musical learning and cross- curricular connections in the arts”, written by Dr Patrick Shepherd from the University of Canterbury New Zealand, study which explores how synaesthetics and the relationship between image and sound forms a useful connection for the cross-curricular teaching of the arts; as mentioned by the author, the terminology used to describe both colour and sound is long-established, with the German term “klangfarbenmelodie” clearly linking colour with musical sound. As we wanted to extend the reach of the concept, we integrated the visualisation not only for sound but for musical motion as well.

Understanding this still at a broader level, we attempted to take a closer look to the relationship between this terms as we try to find a practical application of this relation between visuals, sound and movement.