Data and Machine Learning for Creative Practise — Final Project

6 min readJan 8, 2018

This blog details the result of my final project for my third year module, ‘Data and Machine Learning for Creative Practise’. Using an electromyography (EMG) sensor provided by BITalino and downsampling of a webcam, linear regression is used to translate the movement, gesture, and position of the hand into auditory outputs. The activity of the thenar eminence, the group of muscles on the palm of the hand near the base of the thumb, was recorded to translate this hand position and movement to the corresponding output. The output consists of three combined oscillators, which are then filtered with modifiable cutoff and resonance values to create variation of a simple sound.

Creative Motivation

After receiving a demonstration of two biosensors, the Myo armband and BITalino, I was immediately interested by the use of biological data due to how expressive it is. Using this kind of raw data was something that I had not been introduced to previously and I was excited by both the challenge and the novelty of it. I am particularly inspired by the work of Atau Tanaka[1], even more so after completing this project. His ability to create a fluid and controlled auditory response to movement is something that I hoped to create, although to a much lesser extent, for this work. I now realise that my desired result was extremely optimistic for someone of my skill set, with a limited experience of machine learning analysis and a complete absence of experience in working with biosensors.

Atau Tanaka, STEIM Micro Jamboree 2008

Marco Donnarumma[2] also utilises various biological signals to create precisely choreographed visual and auditory output. The way in which he manipulates the data, for example by overtly exerting muscles as demonstrated in his work below, I find incredibly unique and powerful and enhances the overall result of his piece.

Marco Donnarumma, ‘Hypo Chrysos’

I did not hope or expect to convey the same uniqueness or enhancement in my own project but hoped create a clear perceptual link between hand movement and sound response.

Another work that I found very powerful is an expressive performance piece by Keiran Warner Hunt. The correlation between changes in movement and changes in sound is created by extracting the low frequency content of the muscle activity using the open source XTH Sense[3] project by Marco Donnarumma.

Keirab Warner Hunt, biosensor performance

Implementation

The implementation included using an existing template that receives the sensor data from the BITalino’s serial port[4] and then sends it to Wekinator using the ofxOsc addon.

To improve the signal-to-noise ratio of the sensor input, I calculated the distance between the previous frames value and the current. If this distance exceeded a certain threshold, a one-pole filter was used to remove high frequency content of the signal.

I then experimented solely with the EMG signal testing different supervised learning algorithms. After practising with both dynamic time warping and classification initially, I decided to use regression as:

The EMG signal for the same gesture differed vastly each time. My filter design was not efficient enough to separate the gestures and transform the signal so that movements created consistent or similar values each time. Despite creating multiple training datasets, I realised that the filtered input and time constraints I had would not yield any coherent learning model.
Using classification, for my particular application I felt that this system would be instead more suitable for onsets of auditory events. From the input that I had, I decided that it would be more efficient and interesting to translate the EMG signal into a continuous value due to the limited number of gestures I could create and the difficulty in defining the corresponding signals.

As a result of these tests, I used linear regression in particular in order to try and create a clear auditory link between the movement and positions of the hand and the output.

After deciding that one input feature would not give enough flexibility in creating different outputs, I added more input correlating to a down-sampled implementation of the webcam recording my hand movement. Vectors containing the red, green, and blue values of the down-sampled pixels are sent to the WekiInputHelper and then averaged over a window size of 100 to smooth the values further. These values were then sent to the Wekinator program, which controlled the output of 6 parameters: amplitude, frequency of three oscillators, and resonance and cutoff the filter.

The output program contains a vector of available frequencies of a determined major scale. For the project demo, I used the key of C major. The Wekinator output is transformed into an index for this vector for each oscillator to try and maintain some harmony in the overall output.

Get Becky Johnson’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Unfortunately, my computer could not run WekiInputHelper, Wekinator, the input program, and the output program whilst recording the process using Quicktime so mobile footage had to be used instead.

Evaluation

Before beginning this project, I was very ignorant to just how difficult it would be to map the value of the EMG data to certain gestures. There are many things that I have learnt through this process and so I would do differently if I were to start this project again:

Spend more time on the filter design process of the EMG sensor. Although this did take a large portion of the project build-time, if I were to reattempt the project without time constraints I would not attempt any machine learning analysis of the EMG data until I was fully confident that the signals I received for different gestures were notably different and consistent.
Create a system in which input corresponds for exactly one output. I found that the system of a many-to-many mapping resulted in countless incidents of removing training data as it altered at least one feature in a way that was not desired. For audio in particular, the result became very erratic very quickly and so a singular mapping may have been a better method for this project. In general, I have realised the importance of designing a well-constructed feature system that I will apply in other machine learning projects.
The input of downsampling a webcam, although useful in adding more input to utilise as features, was extremely sensitive to changes in the environment and so when saving and reusing the same training data it would offer different results. Arbitrary things such as wearing different clothing or working at night versus the day resulted in different results by the differing values of the down-sampled r, g, b pixel values and so I often had to retrain the learning model. Additionally, placement of the EMG sensor yielded different results too which required tweaking or removing of previous training. If I were to continue this project, I would set up measures to to be able to reuse training data or perhaps reconsider using such subjective input and utilise more consistent ones.

However, I was happy that I managed to achieve some level of difference between the signals of a clenched fist and a flat hand. Although drastically different, the transformation of the signal into definable values proved to be a hugely challenging aspect of the project for myself. I am happy with the outputted sound of the program but do not feel satisfied that I was able to convey a clear relationship between hand movement, tension, and position.

Overall, I feel that I was far too ambitious with my intentions for this project given my lack of experience with biosignals and hardware in general. That said, I am happy with the progress I have made with improving my understanding of machine learning in general and of Wekinator. For this project, I experimented with dynamic time warping, classification, and finally regression and from this I feel I have gained a much better insight of the advantages and uses of them individually.

Build Instructions

WekiInputHelper setup (left), Wekinator setup (right)

To compile the input program, a BITalino device must be connected via bluetooth to the computer and an internal or external web camera must be available. The settings for both the WekiInputHelper and Wekinator tools are shown above.

The ofxOsc addon[5] was used to send and receive messages to the Wekinator tools and so must be placed in the openFrameworks addon folder. To compile the output program, the Maximillian DSP library[6] was used for sound generation and filtering and so must also be included.