How we turned paintings into noise

Posted on Mon 27 June 2016 in fun

A couple of months ago I was asked by Lidija Matulin, an academy-trained artist from Croatia, to help her on a project that she wanted to work on. She had this idea to make paintings that would extend to an additional dimension and include sound. That sounded (tee-hee) cool to me so I decided to help! Yay!

First, here's one of the painings Lidija made as part of this project, entitled "Vi niste pozvani pa izvolite sjesti" which translates roughly to "You're not invited so please have a seat":

The idea

We brainstormed how a painting could be tranformed into sound and concluded that a simple way would be to have a line that scans through the painting and turns whatever it sees into sound. But how to turn colour information into sound? We tried a couple of approaches.

What we tried

The first approach was just to take the average colour value of all the colours under the scanner line, turn the colour into greyscale, map this value into a frequency and produce a sine wave. Here I have to say thanks to Carl Bussey from Sheffield, who now lives in Berlin and writes software that makes music, because he showed me how to play sine waves in Python using pyaudio. Thanks Carl!

This first try didn't produce sounds that were too exciting so we had to make some changes. We split the scanner line into parts and had each part of the line make a sound on its own. This sounded a bit better.

The other thing that was annoying was that all of the sounds the program made always had the same length, which was dull. So, we made the length of the sound being played also depend on what was in the painting. Here we also decided that it would be better to have two different sources of sound from the painting; one from the background and one from the foreground. As the painting was now split into two different images, it was easy to make the sound of the foreground parts change their length depending on the area that was under each part of the scanner line as it scanned throught the painting.

One more thing that we did was to try different representations of colour other than the usual RGB representation before mapping the colour into the frequency of the sine wave being played. So we tried different combinations of hue, saturation, and value from the HSV representation.

In the end we had a script written in Python with a lot of parameters to play with that produced different sounds for different things that appeared in an image. Once you ran the script given a foreground and a background image, a video would be produced with a scanner line turning whatever it saw into sound.

So I gave this code to Lidija and she tried all kinds of different combinations of parameters and she figured out what to paint to produce what kind of sound and

The final results

The final paintings were shown at an exhibition under the title "Jednosmjernom kartom, umrljanom marmeladom, kroz zečju rupu" / "With a one way ticket, with stained marmelade, through the rabbit hole" in the Center for culture in Čakovec, Croatia.

Here's two photos from there, taken from here: