Sounds

Working with sounds and speech in P5.js

Lesson Goals

• Detect Sounds in P5.js
• Make Sounds in P5.js
• Text to Speech
• Speech to Text
• Sound-Based Services That IFTTT Can Access
• Get Inspired By Sound-Based Projects

Sound — Sensors: Microphones, Sound Level, Speech Recognition, Acoustic, Doppler, Actuators: Assistants, Speakers, Buzzers, Chatbots
Reference — Periodic Table of Spatial Computing

Detect Sounds in P5.js

Step 1 — Set Up the Microphone Input

1. Go to your P5.js web editor and start a new sketch.
2. Create a variable for your microphone input at the start of the sketch. let mic; We declare it at the top of the sketch to make it accessible throughout the entire sketch.
3. In the setup() we want to create a standard canvas size like usual createCanvas(400, 400);.
4. Next in the setup(), we want to create an instance of the p5.AudioIn class and assign it to our mic variable. This object represents the computer’s microphone input. It allows us to access and work with audio from the microphone. It should look like this mic = new p5.AudioIn();
5. Next we want to start the microphone input. To do this we will use mic.start(); which begins listening to the microphone input, and you can use it to retrieve audio data. Steps 1–5 should look like the code below:

let mic;

function setup() {
  createCanvas(400, 400);
  mic = new p5.AudioIn();
  mic.start();
}

Step 2 — Create the Circle Based on Volume

So now we are getting the mic input from our computer, but how do we know that this is working and how do we use this for our sketches? Lets create a circle that adjusts to the mic level to see this in action! For this section we will move to the draw() section of our sketch since we will need to actively adapt our sketch based on the sound level at every millisecond

1. background — In each frame, we start by setting the background color to a black to clear the canvas, ensuring a fresh frame for our dynamic visualization. background(0); You can make this any color.
2. getLevel — Next we want to get the microphone input level. We retrieve the current volume level from the microphone input using the getLevel() method of the mic object. This volume value ranges from 0 (silence) to 1 (maximum volume) and represents the audio input.
3. map — Next we want to map the volume to circle size with the map() function. The map() function takes the volume as input, maps it from a range of 0 to 1 (the volume range) to a new range of 20 to 200 (the circle size range). This step allows us to adjust the circle’s size based on the loudness of incoming sounds. let circleSize = map(volume, 0, 1, 20, 200);
4. ellipse — Finally, we’ll use the ellipse() function to draw a circle on the canvas. ellipse(width / 2, height / 2, circleSize, circleSize); The circle’s position is set at the center of the canvas “width / 2, height / 2”, and its width and height are determined by the circleSize variable which we created in the last step. As the volume of the microphone input changes, the circle will dynamically adjust its size accordingly.

All of the above steps when added to your mic input setup should look like this:

let mic;

function setup() {
  createCanvas(400, 400);
  mic = new p5.AudioIn();
  mic.start();
}

function draw() {
  background(0);

  let volume = mic.getLevel(); 
  let circleSize = map(volume, 0, 1, 20, 400); 
  ellipse(width / 2, height / 2, circleSize, circleSize);
}

🧠🧠🧠CHALLENGE🧠🧠🧠

• How might you change the background color, the circle, or other aspects of your sketch based on the volume level? Hint — see the if else statement below:

    if (volume > 0.1) {
    background(255, 0, 0); 
  } else {
    background(0); 
  }

Make Sounds in P5.js

Step 1 — Write code for loading a sound

1. Go to your P5.js web editor and start a new sketch.
2. At the very top of your sketch we will define a variable to hold the sound: let goatSound; You’ll notice we can name this variable anything. We’ve called it “goatSound” so that we can remember its the sound of a goat. We put loading functions at the very top of your sketch because loading will happen once at the very beginning of the sketch runtime so that these files are available for the later steps of your sketch.
3. Next we want to use the preload() function to load the sound file. These two steps should look like this:

let goatSound;

function preload() {
  goatSound = loadSound('goat.wav'); 
}

Step 2 — Upload A Sound File

1. In your sketch, in the P5.js web editor click the > arrow on the lefthand side of your sketch to reveal the files of your sketch. Here you can see the HTML, Javascript and CSS that makes up your sketch.
2. Click the dropdown menu next to “Sketch Files” and select “Upload file.”
3. Let’s upload a small sound file to get started. You can download a simple sound file here goat.wav and then drag and drop this file into the upload window. You will then see this file appear as an additional file in your sketch folder.

Step 3 — Write Code to Play the Sound File

Ok great, so we have a sound, and we’ve loaded it. But now we want to hear it. So we need to write code that tells P5 to play the sound.

1. In the setup() function, play the sound when the sketch starts:

function setup() {
  createCanvas(400, 200); // Set your canvas size
  goatSound.play(); // Play the sound when the sketch starts
}

Step 4 — Run Your Sketch & Set the Volume

1. Click the “Play” button at the top of the code editor to run your p5.js sketch.
2. You’ll hear the “goat.wav” sound file play automatically when the sketch starts. If you don’t hear it but don’t see any errors it may be too quiet. Let’s play with making it louder.
3. You can change the volume by using goatSound.setVolume(10); before the goatSound.play();. In p5.js and many other programming languages, the dot (.) notation is used to access and manipulate properties and methods of objects. So when we have created this variable goatSound that represents our sound file, we can use the dot notation to manipulate the properties of this file. Here 10 represents the volume level. Try changing this to 0.5 or 20 to see it change.

Your full sketch should look like this:

let goatSound;

function preload() {
  goatSound = loadSound('goat.wav'); 
}

function setup() {
  createCanvas(400, 200); 
  goatSound.setVolume(10);
  goatSound.play(); 
}

That’s it! You’ve created a simplified p5.js sketch that plays the sound. Remember to save your project in the web editor so you can access it later.

🧠🧠🧠CHALLENGE🧠🧠🧠

• How might you create a button to play the sound?

Bonus — Change the Pitch

• We can then do fun things like change the pitch based on the position of our mouse. Here’s an example!

let goatSound;

function preload() {
  goatSound = loadSound('goat.wav'); 
}

function setup() {
  createCanvas(400, 200); 
  
  goatSound.setVolume(10);
  goatSound.play(); 
}

function draw() {
  background(0);
  let mouseXNormalized = map(mouseX, 0, width, 0.5, 2.0);
  goatSound.rate(mouseXNormalized);
  
  fill(0);
  text('Pitch: ' + mouseXNormalized.toFixed(2), 20, 20);
  }

Bonus — Change the sound…maybe?

Since you probably aren’t interested in making a contraption that sounds like a goat…you’re probably wondering “How can I load a larger sound file and one of my own files.” To which I respond, “Great question!”

Get Violet Whitney’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In order to do this, you’ll need to run P5.js locally on your machine and run a local web server. This is quite common in software development and used during the development phase before hosting code projects on the cloud. We’ll do this in a future tutorial. For now just be satisfied with with “goat.wav” and the varieties of projects that can be done with “goat.wav.”

If you’re adament about using other sounds, you can find your own sounds but they’ll need to be smaller than 5mb in order to be hosted on the P5.js web editor. You can find many sounds online here: https://freesound.org/ You’ll need to create an account and most files will need to be downsized.

Text to Speech

Getting A Computer to Speak

P5 can also speak by accessing the built-in Web Speech APIs available in modern browsers.

• SpeechSynthesisUtterance is used to turn a string into something that can be spoken. This is where you would add your unique message as a string.
• You can set additional properties like volume, rate, and pitch if needed.
• window.speechSynthesis.speak(message) is used to initiate the speech with the specified message.

NOTE — Browser support for the Web Speech API varies; test in different browsers. Ensure the P5 web page has proper microphone and speaker permissions.

function setup() {
  let message = new SpeechSynthesisUtterance("hello"); 

  // Set additional properties (optional)
  message.volume = 1.0; // Volume (0.0 to 1.0)
  message.rate = 1.0; // Speech rate (0.1 to 10.0)
  message.pitch = 1.0; // Speech pitch (0.0 to 2.0)

  // Speak the message
  window.speechSynthesis.speak(message);
}

You can learn more about the built-in Web Speech APIs available in modern browsers here.

You can also do more elaborate things like this:

function setup() {
  createCanvas(400, 200);
  textSize(32); // Set text size
  textAlign(CENTER, CENTER); // Set text alignment to center
  
  synth = window.speechSynthesis;
}

function draw() {
  background(0);
  fill(255, 0,0);
  ellipse(mouseX,mouseY,10,10);
  fill(255, mouseX, mouseY);
  text('hello' + mouseX, mouseX, mouseY);
  pitchMultiplier = map(mouseX, 0, height, 0.5, 2.0);
  rateMultiplier = map(mouseX, 0, height, 0.5, 2.0);
}

function mousePressed() {
  let utterance = new SpeechSynthesisUtterance("dog" + mouseX);
  utterance.pitch = pitchMultiplier;
   utterance.rate = rateMultiplier;
  synth.speak(utterance);
}

Speech to Text in P5.js

The following uses the browser’s webkitSpeechRecognition to transcribe audio from the mic whenever the spacebar is held down and draw it as a P5 text().

//note you must first click the canvas with your mouse before holding down the space bar

let speechRecognition;
let isListening = false;
let textToShow = "";

function setup() {
  createCanvas(400, 400);
  background(255);

  // Initialize speech recognition
  speechRecognition = new webkitSpeechRecognition() || new SpeechRecognition();
  speechRecognition.continuous = true;
  speechRecognition.interimResults = true;
  speechRecognition.lang = 'en-US';

  // Handle the result event
  speechRecognition.onresult = function(event) {
    if (event.results.length > 0) {
      textToShow = event.results[0][0].transcript;
    }
  };
}

function draw() {
  background(255);
  fill(0);
  textSize(32);
  text(textToShow, 10, height / 2);
}

function keyPressed() {
  if (keyCode === 32) { // Spacebar
    if (!isListening) {
      speechRecognition.start();
      isListening = true;
    }
  }
}

function keyReleased() {
  if (keyCode === 32) { // Spacebar
    if (isListening) {
      speechRecognition.stop();
      isListening = false;
    }
  }
}

Sound-Based Services That IFTTT Can Access

• Alexa
• Google Assistant
• Spotify
• Sonos speakers
• SMS and calling

Audio Projects for Inspiration!

Bots & Speech

As cynical start-up idea generator, Darius Kazemi
As poetry maker, Allison Parrish
Gender, Kate Compton
Magic Realism, Kate Compton