Knitting into Sound

This week I focused on the sound design. I created a quick prototype that I could connect to Max MSP. While I could see patterns in the incoming data as I knit, it was difficult to make sense of it musically.

Arduino Feather with MPU6050 Accelerometer + Gyro

One approach was to try to smooth the numbers in Max. I worked with an ITP Resident to map the numbers, smooth them, then clip them. I mapped the different accelerometer values to the gain of different frequencies. Each axis (X, Y, and Z) were assigned a specific note to make up an E flat major chord. Based on how much the hand was moving in any direction, the corresponding note would be louder. The sounds were still very erratic. Though some patterns could be heard, it was certainly not what I would consider rhythmic.

I started thinking about how the sound design would come together using both accelerometer and gyroscope data. I thought if I could use the accelerometer to control the volume of different notes, then I could switch between different notes using the gyro. One way to do this is through midi. I could map the values coming in to notes on the midi keyboard. I tried this out with only one gyro axis, but it only seemed to add to the chaos.

A resident helped me find the appropriate ranges to deal with the numbers coming in, but it was still very difficult to make sense of them. I also probably wouldn't be using Max in the final version, so it was important not to rely on it too heavily. In Arduino I was able to create samples of the numbers and average their change over time. Suddenly I had really clear numbers. When my hand was flat and still on the table I got consistent zeros, and as I moved my hand around I could watch the different axis values grow and shrink consistently.

Oscillators vs. Midi

One major decision I've needed to make is whether to embed my audio into the garment. I have been strongly encouraged not to, and instead to send my data to a computer that could map to Midi values. The motion data would be sent to the computer via bluetooth or wifi, then the signal would need to be sent back to the garment using a bluetooth speaker. The benefit of sending my values to a computer would be a much better range of sounds. I'd be able to compose samples and switch instruments, because I could work directly with Ableton Live. This would ultimately sound better for the purposes of documentation because I could feed the actual sound into the video instead of recording it through a mic.

The phrase I kept hearing in favor of external audio was "the quality will be better", but that was an abstract concept to me because I am so new to audio. Recently I found an example of sending sensor data via wifi to a computer for musical expression. Imogen Heap collaborated with a team to make a pair of gesture sensing gloves for musicians. This example helped me really grasp the level of fidelity I would be missing out on with embedded audio.

The gloves that will "change the way we make music", with Imogen Heap from Dezeen on Vimeo.

So, why did I chose to embed my audio? Embedding the audio is actually important to me. For one, I want to maintain the integrity of an enclosed system. I am focusing on the relationship between the wearer, their garment, and the knitting process, and that narrative. With embedded audio, the wearer has complete freedom to move around, untethered to an external machine and its software.

Secondly, I really like hardware. During my time at ITP, I have spent a lot of time developing my hardware prototyping skills. I have spent countless hours becoming a resident expert on soft circuitry and wearables. My thesis is an opportunity to focus that knowledge into a project and, honestly, show off what I've learned. Not embedding my audio at this point would mean minimal hardware, and a lot of room for sound design – a skill I have no prior experience with.

I have also chosen to work with embedded audio simply because it's possible. This was actually a big factor, because I immediately got feedback that it might not be. In making a decision I did a lot of research into current technology on the market. A few people recommended that I look into the Teensy, although none of them had ever tried to use one. The Teensy has a audio adapter board made specifically for the type of project I wanted to make. It's very small, runs Arduino, has its own audio system design GUI, and can run on a 3.7V battery.

Teensy 3.1 Teensy 3.1

Teensy audio board Teensy Audio Adapter Board