Rock Opera (Week 9 Theme: Play & Materiality)

***This project was made in collaboration with Benjamin Mansur & Emmett Redding.***

Theme: Play & Materiality

Context:

When Matt was going through his lecture, I was really inspired by one of the works he showed - the OutsideIn Capsule (MaudWongDesign 2014) by Mengyang Wang (ex-MAGI student) - which was an interactive sound installation that provided users an audio experience in an isolated dark space through tactile interaction. It reminded me of the soundscapes I often listened to in Calm, a meditation app, and how it transports me to these tranquil and peaceful worlds through audio when I close my eyes.

I had this idea for an interactive experience where touching objects would trigger audio soundscapes for the player; for example, a seashell can play an ocean soundscape, or a leaf can play a forest soundscape. The objects would act like portals to connect the physical and digital worlds. Having previously chatted with Ben about collaborating for an APD exercise, he also liked the idea and we decided to collaborate. Ben also invited Emmett into the team.

Through this installation, I hope the user can experience the following pleasure categories of play (Costello and Edmonds 2007): exploration - where participants will be interacting with and explore each object's soundscape; captivation - where participants will hopefully feel mesmerised by the soundscapes and take them back to nature; and sensation - where participants can enjoy physically touching the texture of each object.

Method: 

Using the Makey Makey kit, create a tactile, interactive piece where different nature soundscapes will play when the user touches physical objects.

Response: 

While I got started with setting up the Makey Makey and programming in Processing, Emmett and Ben got to work trying to find small physical objects we could use. It was a challenge as we didn't have many tools or conductive materials we could shape and utilise (e.g. aluminium foil, conductive pens, etc.) and only some used wires that were lying around the MAGI studio. We originally wanted very specific objects like a seashell that would trigger ocean sounds, but with limited time and location, it was hard to source these site-specific objects.

After Emmett and Ben went scavenging outside, they found some rocks outside Building 7 that were suitable as a first prototype. Having rocks rather than specific objects also created an element of exploration in terms of play, as players are invited to touch each one to find out what the hidden soundscape was for each rock. Each rock also provided a unique shape, colour and texture to the piece - like they had a story to tell - so I was happy to go with this idea.

Emmett started sourcing sounds, Ben got to work wiring up each rock and finding visuals, and I continued debugging code to make things work. Each rock was then "wired up" so when the user touches the rock and completes the circuit with their hand, a soundscape would play and a visual image would display. We ran into some issues with loading times as the images and sounds were quite large in file size; after fixing this, we got Evan to playtest. It was interesting to see how he wanted to play multiple soundscapes at once. After observing his actions, we tweaked the model so that multiple soundscapes can be overlayed on top of one another, and got rid of the visuals to leave it up to the players' imagination to visualise each soundscape. As Rueb (2004) argued, audio is a more appropriate medium for engaging with natural locations... rather than being distracted by screen-based visual content.

We then playtested again with Hizi. After experiencing Rock Opera, he also suggested for us to look at A Soft Murmur (Brady n.d.), a website that plays ambient audio loops with adjustable volumes so users can mix their own ambience, and suggested that we implement some of these functions in our next iteration / prototype.

Credits: BBC Sound Effects Library for soundscapes; Pixabay for visuals.

Reflection:

Overall, we felt that having the ability to overlay soundscapes definitely added to the feeling of ambience for the players. We also thought to include shorter and sharper sounds (like a bird call, or a thunder roll) so players could clearly hear the audio feedback from their tactile input. At the moment, the soundscapes kind of just blur into one another, and they're quite long, so it takes a while before they finish playing. It was really interesting to see players interact with Rock Opera, especially when they started playing the rocks like a keyboard or synthesizer.

Upon reflection, I also thought of a couple of tweaks that could've improved the experience given more time:

• Having soundscapes from different time periods - one suggestion I made was a dinosaur soundscape
• Allowing the user to fade in / out the soundscapes, either by pressing and holding the input on the rocks, or pressing the rock again to pause the soundscape
• Inscribing symbols to the rocks that represented the soundscape location, like rock paintings
• Try the same experience using objects that actually related to the locations of the soundscapes

In terms of the collaboration experience, it was good to split off the tasks of programming, building and sourcing assets, so I wasn't having to deal with everything at once. However, I found it quite stressful when everyone else in the team was waiting for me to debug the code and fix issues on a limited timeframe. It was moments like this where I would prefer to code in a quiet environment without this added pressure and distractions from other students, as I found it really hard to focus and think logically. I felt like I was letting the team down as I was so slow at programming. However, I really enjoyed the way we collaborated and utilised our strengths - such as my experience with Makey Makey and Processing, Emmett's sound editing experience, and Ben's quick problem-solving skills.

This idea of layering sounds also relates to my studio project where sounds of individual musical "creatures" add to the base musical track that plays. I also found this week's theme of the material and immaterial highly relevant to my project - as I'm capturing the physical movement of bodies as data that drives the actions of the game's characters. I find this concept of translation between the digital and physical worlds fascinating, and I hope to continue exploration into this field through sensors and alternative controllers in the future.

References:

Brady, G n.d. A Soft Murmur, viewed 26 September 2019, <https://asoftmurmur.com/>.

Costello, B & Edmonds, E 2007, 'A study in play, pleasure and interaction design', proceedings of the 2007 International Conference on Designing Pleasurable Products and Interfaces, pp. 76-91.

MaudWongDesign 2014, OutsideIn Capsule, Cycling '74, viewed 26 September 2019, <https://cycling74.com/projects/outsidein-capsule>.

Rueb T 2004, Teri Rueb's four questions, Mobile Audience, viewed 26 September 2019, <http://mobileaudience.blogspot.com/2004/12/teri-ruebs-four-questions.html>.