Hi, I am aline martinez.
I make
art
using data.
I foster
innovation
.
Sonification
The Sonification is a costume-controler which eagers the visualization of the dancer’s motion through sounds. The project places the body as central input. Every movement of the arms’ and legs’ articulations are used to modulate the soundscape in a space. The costume is an interactive dispositive with sensors strategically placed on the wrist, elbows and knees.
The sensors were crafted using the knitting technique with textile materials. Tests using crochet, knitting and embroidering methods demonstrate that the circular-knitting one senses the body accurately. The final costume features 6 sensors connected to a backpack containing an LilyPad with 6 voltage separating circuits, batteries and a low-power digital radio (Xbee) for communication with the external computer. Also, an application written in the Java processes the signals in real time and creates sound output through a commercial music sequencer.
Bellow, the process of making it!
.
.
.
Context
The challenge during the class of Human Computer Interaction at the Bauhaus Universität was to craft interactive wearables for dancers. The professorship had a partnership with semi-professional dancers from Altenburg, with whom we could test our projects.
Research
Interactive costumes, so far, usually visualize actions with visual methods, such as LED lights, video mapping or movable parts. We wanted to try a different approach, namely visualizing with sound. One paper uses the term audiolizing to describe body movements that are transferred into sound. However the word is rather hard to pronounce and the direct takeover from visualizing sounds naive. Therefore we chose to call such a method to visualize body movement in dancing by sound “sonification of dance”. Sonification usually is used to visualize sensor data (for example the click of a geiger counter) with non speech audio.
Wikipedia: Sonification is the use of non-speech audio to convey information or perceptualize data.[1] Auditory perception has advantages in temporal, spatial, amplitude, and frequency resolution that open possibilities as an alternative or complement to visualization techniques.
There exists costumes and sensors that can create music, or sound, from body movements. However these devices are targeted towards musicians and not dancers. Famous British musician Imogen Heap created in a collaboration with computer engineers and other artists gloves that can make music.
With devices such as this one, the movements the performer makes are made with respect to the musical notes, harmonics and beats the musician want to achieve.
Our costume will be tailored for dancers, therefore the sound output will depend on the dance poses, the body movements, the performers have in their mind. In a free choreography both will influence each other and the performance becomes interactive. In a trained dance performance, using prescribed movements from start to finish, we will not experience sound as interpreted by a musician but we will hear the sonified dance movements itself.
The Sensors
The idea was to develop sensors seamlessly integrated to the textile surface. To find the right one, we tested different ways to produce textile sensors to push, bend and streatch. The one I am showing here is the stretch sensor once this one was the one we used. For more, please check ???here???.
Knitted Stretch Sensor
This one is super fun to build and very realiable too. The knitted textile has the property to come back to its initial position when no traction is applied. As the resistance measured depends on the
tension in the fabric, it is important that the fabric contracts again, to ensure that the values measured correlate to the angle of the articulation wearing the garment (and therefore consistency of the signal read).
In a simple test with 333timer, the sounds changes like so:
Test, Iteration
The final costume features 6 strategically placed sensors (hands, elbows, knees) that are connected with conductive yarn to a backpack; this contains the Arduino micro-controller LilyPad with 6 voltage separating circuits, batteries and a low-power digital radio for communication with the external computer. Furthermore an application written in the Java language Processing processes the signals in real time and creates sound output through a commercial music sequencer.
After buiding the costume, on Wednesday, 22nd June, we visited the dancers’ rehearsal a second time. The days before were really busy since we needed to finish the costume prototypes. Arriving there, we were a bit nervous because the technologies did not work properly overall. We fixed cable and thread breaks until we left Weimar. Nevertheless, things worked at least partly and we were curious how the dancers would react when seeing and wearing the costumes. On-site, we video-recorded everything and conducted interviews with every single dancer who wore one of our costumes.
The semi-structured interview with our dancer, who tested the prototype first, confirmed some of our opinions about the costume and gave us also a few new contrasting insights.
One dancer wore the costume and danced three dances with it. One solo performance with just headphone music as guidance, once in the group with background music and once she did the choreography from her head.
The dancer talked in the interview mostly around the two distinct characteristics of the costume: knitted and sound. From the transcript we color coded the responses and grouped them into the categories of our research question proposals. In the following the most important quotes from the test dancer and from the general discussion, which sometimes also carry over into other categories:
Usability:
“I don’t know if it was because I wore the costume the first time, but because of these fibres I was a little careful, also because of this technology in the back.”
“I think I could put it on myself, but this time I didn’t want to break it.”
“I could not raise my arm all the way up and I did not want to force it.”
“But if more than one dancer has the costume and they are not perfectly synchronous, you can hear that easier, especially because we are not so professional yet.”
Wearability:
“I liked that I could move very freely.”
“It followed the movements, did not disturb.”
“I did not like that it got extremely warm. [..] If I had to dance for two hours, and the headlights were on. [..] It would get uncomfortable.”
“Maybe not from wool, so that you feel comfortable to spend more time in it and explore it.”
Effect:
“Reminds me of such very modern fashion that you see on shows, very abstract.”
“I would say cool, because with the wool it is different and something completely new.”
“I am not sure how well it works on the big stage. The wool is all black and has many details. [..] This you maybe cannot see from far [..] It might look like a wool hank.”
“The costumes are all very futuristic, […] they fit mostly the modern dance as you don’t always have a tutu and some of the dances are silent or they explore with breathing. “
“The knitted suit was very puristic and if you imagine that these things on the side would grow even more, as you said, that transports an interesting message. “
Sound:
“It influences other senses than the visual one, this you don’t have yet with dance. It brings a new dimension.”
“In the beginning I could not hear so much influence [..] but when I got down with my knees I could hear it with the headphones on.”
“The dance I did is actually quite happy but the sound from the costume is not, maybe something more melodic [..] or for a different character, a powerful one.”
“I did not expect that It would make these constant sounds, like from an akkordeon. Later it also made other noises.”
“Sound is something from the future. It could be great to hear a whole dance. “
“It would be great for blind people.”
Choreography:
“Maybe some dance with maximum movements, just fit to the costume and its sounds. A choreography with only sounds from costumes would be great, so that you as a dancer create the music yourself”.
“I think multiple different costumes would sound great.”
“It is not easy to integrate into existing choreographies, you need a muted dance that is developed just for the costume, but it would be an interesting approach. Would it actually be possible to have melodic sounds or other sounds with the costumes?”
Interpretation
Usability: The costume can creates new challenges for dancers.
Wearability: The costume is flexible but too warm.
Effect: The look of the costume is suited for dance.
Sound: The sound should be more direct and friendly for this choreography.
Choreography: It needs its own choreography to show the full potential.
The general impression about the costume seems positive. The time the dancer had with the costume was very short and the costume was not tested with an audience which is why some of the very interesting questions the sonification costumes rises will be left unanswered. However our one dancer and the dance teacher reported that costumes that create sound are something very interesting that should be worked with. It is not just for blind people but adds a whole new dimension that choreographers might be very interested in.
We found that such a costume is not something for amateurs as the sounds and choreography need to be adjusted hand in hand if the costume should just not be an instrument that creates arbitrary noises. From the point of view of the dancer the negative thing about the costume is that it is too hot to dance in. Even if the dancer tested it on a particularly hot summer day traditional spotlights also create a lot of heat in winter.
Our opinions is that the costume shows that sensors can be integrated into the strings of fabric but it must be made professionally with a machine so that other fibres can be used that are more suitable. On the other hand such a knitted costume might fit more into theater where characters do not move as much. Obviously the set up with the computer and speaker takes time but this time can be scheduled before a performance.
Such a costume (and all interactive costumes) cannot just be hand over to dancers without readjusting choreographies as it changes the meaning of a dance and the importance of the dancer. We suggest to create interactive costumes that are specifically suited for training: washable, light, robust, easy to repair, comfortable, easy to put on and easy to maintain. This is the only option for choreographers and dancers to create a choreography specific for the costume, which should be worn only during the performance.
Furthermore we conclude that not all characteristics of an interactive costume can be optimized. Functionality, usability, wearability, effect and integration do not go completely hand in hand and the dancers, choreographers and tailors have to balance their preferences.
Wisp Festival Digital Arts, Leipzig, 2016
SonificationSmartTex, Weimar, 2016