Coghack #28: IIIF Audio Zooms
Immersive Avian Soundscapes in Zoomable Images
The focus of this hackday was experimenting with the International Image Interoperability Framework (IIIF). We decided to explore the idea of audio located within a zoomable image. We wanted to make the sound an immersive part of the experience, dynamically changing as the user panned and zoomed. A few ideas of the kind of content that this would support came to mind: A crowd scene where you could move in closer to pick out individual conversations, or an image of an orchestra or band with accompanying music track where the mix of instruments would vary as you navigated the image. For our hackday example, we decided to go with birdsong. This would work well as the short tracks could loop independently and sound natural together, allowing users to create a recognisable soundscape and identify individual birds as they zoomed in.
For our image we were looking for something that would work well with distinct areas to locate the sounds, ideally representing a single habitat. We sourced an image from the Park and garden birds guide illustrated by Chris Shields and published by the Field Studies Council. The birdsong audio was taken from the extremely helpful R.S.P.B bird identification guide.
We loaded our image onto a IIIF server and used OpenSeadragon as our zoom viewer, modifying it to add the required functionality. The individual birdsong tracks were played through separate audio elements on the web page but so far not anchored to a location in the image.
Get cogapp’s stories in your inbox
Join Medium for free to get updates from this writer.
To achieve the effect that we wanted of adjusting the volume of track based on the apparent closeness of its source location in the viewer, we needed a solution that would create the impression of moving in a 3-dimensional space as we zoomed and panned a 2D image. By locating our audio sources at particular co-ordinates in the image, we could have used the zoom level as a z-axis to produce a 3D vector that would give a virtual distance with which to adjust the volume. With a few other considerations taken into account, this all seemed like a little too much maths for a Friday and so we went for an alternative proxy approach. We defined a hit area rectangle for each sound on the original image, and the amount of the viewport occupied by this rectangle would determine the track’s volume. When zoomed close to a location the hit area could occupy 100% of the screen, playing the track at 100% volume. As we zoomed out, the apparent size and volume would decrease. Similarly, when panning, as more of the hit area moved onto the screen the volume would increase. If the hit area were entirely out of view the volume would reduce to zero. OpenSeadragon helped us out here as it has methods available to make working with such hit area rectangles fairly easy.
We wrote the code with just a couple of birds initially, but it wasn’t until all the audio tracks were loaded and mapped that we fully appreciated how the piece would come together. The result was very pleasing, offering the exploratory experience we had hoped for.
One addition that could enhance the immersive feel was stereo sound. With the help of an article on MDN, we were able to leverage the Web Audio API, and map the stereo pan of each track between the speakers as its corresponding location moved left to right on the screen we were able to make it feel, particularly with headphones, as if you were poking your head right into the hedgerow. We determined this by simply finding the midpoint of the intersection between the hit area and the viewport and applying that proportion to the audio pan scale.
We were pleased with the end result and have put it online here in the hope that you enjoy it too. With the constraints of the hackday we were focused on just one particular implementation and it would be interesting to see how well this translated to other content. It may be that the hit area approach is not sophisticated enough for a general purpose implementation and we would need to use a more accurate spacial model for setting volume and stero pan. Rather relying solely on looping audio, perhaps some implementations would benefit from a trigger distance to begin playback. In this instance we hardcoded our audio and positional information into the web page, but for a more flexible approach we could add this data into a IIIF manifest, along with other metadata that could be interpreted by the viewer.
Steve and Jon are developers at Cogapp, a digital agency specialising in the cultural sector. Please get in touch if you’d like to hear more about our work.
If you’re interested in joining us for a hack day, there’s more information on our website.
We’re on Twitter or you can contact us via our website.
