Sound Synthesis

Body Horror Instrument

This week I had the chance to explore sound synthesis in p5.js with Ibrahim. We were both interested in the idea of creating our own instruments. Given the creative freedom with this project, we imagined an unconventional instrument which subverts our typical associations between body parts and sounds.

We wanted to incorporate interaction into our sketch using ML5 and PoseNet, so users could use their own body parts to play the instrument. We thought it would be best to work on two separate sketches – one for the skeleton and another for the body parts and sound effects that we would later merge.

We were keen on creating a skeleton for the body based on Dan Shiffman’s tutorials. The trickiest part of PoseNet was remembering that each pose has unique key-points. We decided that the sketch should focus on the users skeleton and not have an overlay of the webcam capture. To make sure the user could still see different limb points, we colored the nodes yellow. The wrists were made blue and the nose red, to get rid of the color monotony. Here is a link to our skeleton sketch.

We started working on getting the body parts (images) and sound effects into p5.js with a separate sketch here. We decided on having each limb be a button that plays some happy sound effects. when the user touches with their PoseNet skeleton.

Unfortunately this part of the two sketches failed. The size and pixel quality of the images coupled with the number of sound effects kept causing p5.js to crash. The sketch could only run a single image with sound before breaking on us.

Body-Horror Instrument

We both had fun with this idea and would love to continue working on it together while at ITP.

Sound Oscillator

Having spent most of the week on this and seeing it not work we decided to scrap this idea and make something else. Ibrahim had a great backup plan to create a simpler version of the instrument of our dreams…a xylophone. We constructed the xylophone using tutorials from Mister Bomb. The sketch itself uses oscillators and an envelope with ADSR to create the sound rather than uploading and using pre-recorded sound effects. Here is a link to our sketch.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s