When I started my walk down the dark and terrifying path of coding, I did so because it seemed like the only way to link certain applications was through running scripts I found on Github. I remember feeling like a hacking god the first time I got video output from Ableton visual noise generators to show up in the same Processing sketch as my Resolume output. And while my professional path has led me far from my early days of gluing disparate applications together, I still find myself approaching coding as a bridge in many cases.

In several artforms, there’s a kind of floating sense that nothing is truly original. The simple, great ideas have been spit out and regurgitated over the years, and ‘modern’ trends tend to recycle and alter ideas from 20 years prior. While I don’t think this is entirely true, there’s a nugget of valuable information there: So many great ideas are out there already, building blocks laid down by giants before us, and we can use them to build bigger and better things.

Take Ableton and Resolume, for example. The wonderful people at Resolume saw fit to build out what interaction between the two live performance apps they could, but at the end of the day, creative use of Processing, Max 4 Live, and Syphon open up an entire universe of creativity and flexibility that could never realistically be covered efficiently in the core application.

Similarly, with Algorhythm, I didn’t create some brilliant new relational algorithm, but instead simply found a way to twist Spotify’s wonderful API to provide a service that I found myself wanting — a twist on the recommendation functionality that Spotify already had, but with more direct user control. And I’ve found myself using the functionality time and time again.

As a new developer, we are frequently told to find a compelling project to work on as a way to learn quickly. For me, independent projects have been absolutely crucial to my learning. However, many people I know seem to get blank-slate anxiety — when faced with the whole world of potential projects, they get freaked out and wind up building yet another task list manager.

If this sounds like you, consider building a bridge instead of building a whole town. Instead of building some elaborate new suite of functionality, and expecting yourself to Steve Jobs some brilliant idea into existence, simply extend the functionality of an api or program, or build out some link between applications that you like. Don’t reinvent the wheel, hop on the backs of some giants instead and go for a ride.

And now for some more bridges.

Before I picked up production, I played a variety of instruments. While I was never any kind of decent soloist or improvisational player, I always loved the instant feedback and freeing experience of just jamming. Coming in with nothing planned out, playing a lick, and seeing where it takes you. As much as I loved music production when I started learning, I constantly sought for that level of immersion in my music.

Enter Ableton Live. Ableton is a DAW (Digital Audio Workstation) built around the workflow of old-school samplers and sequencers. Rather than laying out files or strips of MIDI in a linear fashion, like Pro Tools, Logic, or Reaper, you can construct a grid of midi or audio loops, which can keep themselves in time and fire at the tops of measures, so you can coordinate and fire and entire song, in realtime, by yourself. This workflow made sense to me. Suddenly, I could fire a drum loop, improvise some guitar over top, immediately loop it, drop another backing part in the adjacent track, loop that, kill the drums and start a new beat for the break, etc… all in almost-realtime.

So when I finally got my LED matrix up and running, and could syphon video to it at will, my first instinct was to find a way to make designing visuals for it as intuitive as constructing music in Ableton — and what better way than by leveraging Ableton itself?

All major music software & hardware these days communicates over MIDI. MIDI is a protocol for communicating musical information — a channel can transmit 127 notes, each with a velocity (how hard you hit the key) of 0–127. It can also send CC (Continuous Controller) information, like knobs and faders, with a value of — you guessed it — 0–127. Since this is a nearly universal standard, it’s not uncommon to find all sorts of interesting projects leveraging MIDI. So how hard could it possibly be to rope MIDI into Processing?

Turns out, not hard at all. MIDIBus is the Processing library that opens the doors to MIDI communication in Processing. By routing MIDI out to a virtual bus in the computer (Mac has built-in utilities for this), MIDIBus can grab the information and pass it in as a series of integer values that we can then put to work within Processing.

My first task was to get something on the screen. I set up an array to store all the incoming data. The index would reference the note number, and the integer stored would reference the incoming velocity. Within my draw function, I simply iterated through whatever notes I wanted to render an image and set the alpha of the RGBA to whatever the incoming velocity was. Additionally, some funky mapping and math put each of the 64 notes I wanted to display on an 8-by-8 grid across the entire screen. The obvious problem with this, though, is that once the notes are up, they’re there to stay.

So I included a decay value for every note — each frame, it would decrease in value by a set amount till it reached zero. Cool. Now we’re getting somewhere. Next, I set up more storage for CC values. Since Ableton typically operates in sets of 8 knobs, as does the MIDI hardware I prefer, I can make this a much smaller array. To get some more hands-on control, I mapped one knob to control decay rate, and three to control the R, G and B values of the orb color. Now we’ve got some flexibility. Hitting record and playing with the buttons and knobs lets us loop and modify our orb visuals fluidly, in realtime.

Following this logic, I’ve continued to build out a massive processing sketch, using MIDI rolls and drum racks in Ableton to control everything from simple full-screen color burst effects, to enabling / disabling audio reactive programmed elements. After a little bit of practice, hopefully, I’ll be able to bring Ableton along to a DJ set and play the visuals along with the music, live. And the beauty of it? Because of Ableton’s modular nature, I can save down midi clips for specific songs if I so desire, and fire them in whatever sequence I want, if I want something more complex than I can realistically do with two hands.

So, to conclude this four-part adventure into the weird world of mixing Processing with everything under the sun, I’d like to say this: Code is a tool. A phenomenal tool. At the end of the day, all you need is a good idea and some baby steps in the right direction and there’s some way to get there. Before starting this project, I’d never touched Java. I’ve still never touched it outside the context of Processing. Despite that, I’ve been able to wrangle it into the rough shape of a program I could have only dreamed of three months ago.

Get out there and make something.

So while music production, like coding, has become more accessible than ever before, the body of knowledge available can be outright terrifying to a new student. On top of that, both fields are awash with talent — everywhere you look, you see examples of masters of their craft outputting projects drenched in subtlety, nuance and grace. How can our hapless student ever hope to hold her own work up against such perfection?

The book that started me down the path to answering this question was The Art of Learning, by Josh Waitzkin. The book tells of his journey from one of the world’s finest chess players, to one of the world’s finest Tai Chi competitors, and how he approaches learning, specifically. I picked this book up on the recommendation of Sean Plott, the man who made Starcraft accessible to the American masses. He recommended it as a path by which to refine Starcraft play — an incredibly complex, competitive, and mentally demanding game. Although my Starcraft play is still mediocre at best, the concepts I picked up in the book have helped me refine my own learning efforts, as well as pass on salient advice to students who hope to reach for the stars.

First, let’s address the fear. In his book, Josh writes, “The key to pursuing excellence is to embrace an organic, long-term learning process, and not to live in a shell of static, safe mediocrity. Usually, growth comes at the expense of previous comfort or safety.” This quote rocked me particularly hard. Suddenly, a master of his craft was telling me, It’s okay to be scared. It’s okay to be uncomfortable. It’s okay to not be a phenomenal talent right out of the gate. So how has this played into my own learning efforts? I have made an effort to embrace the difficult. When you find yourself sitting down and thinking, Yeah, I could do that, but that sounds like a bit much, quell that part of your mind. It is safe, and boring. Reaching for that slightly-crazy idea, pushing yourself to make your sounds a little more perfect, dumping hours into getting that perfect mix, or chopping dozens of not-quite-right Stack Overflow responses into the function you actually need — this is the valuable learning, the learning that pushes you leaps and bounds ahead in the game.

But Alec, you say, What if it doesn’t work? All the better. We learn best from our mistakes. If you nail something on the first try, great! You’ve learned how to do that thing. But if you flail around and struggle with something for hours, try dozens of different versions, attack the problem from different angles, now you’ve learned countless discrete tidbits of knowledge, and a plethora of new approaches to the issue that, while they weren’t the right approach for that specific problem, will be sitting ready in the back of your mind for the next problem.

John attacks another critical stumbling block for many new coders and producers in his book — what to learn? “Depth beats breadth any day of the week, because it opens a channel for the intangible, unconscious, creative components of our hidden potential.” This quote brings to mind a concept I learned about while firedancing — flow. Flow is the mental state of peak creativity and performance. Flow happens when your planning, worrying, judging mind cedes the reins to your momentary, creative, and flexible mind. Flow happens when you relinquish a certain element of control, and it can only happen when a thing becomes second nature — when you don’t have to think about how to do the thing, but merely what you want to create.

While this sounds very artsy and not at all applicable to cerebral tasks like coding or producing, I believe elements certainly apply. Consider the new developer, drowning in a sea of languages, with everyone telling her how she should learn Python — no wait, learn Rust, it’s so flexible! — nonsense, learn C, it’s the best foundation — but Javascript is so marketable — —

It’s traumatizing, honestly. If that new coder goes off and tries to learn a dozen different languages, she’ll struggle with any piece of code, regardless of what language it is in. But if she picks one — regardless of which one — and dives deep, until that language becomes an extension of her mind, and she can almost unconsciously produce functional and conventional code, all that remains is deciding what to make, instead of wasting time worrying about the syntax and conventions. Similarly, in production, learning two dozen synths leaves the new engineer proficient in none, whereas picking one and sticking with it will let her build almost any kind of sound.

Finally, there’s the oft-touted concept of ten thousand hours to mastery. At first glance, that phrase seems absolutely terrifying, but after spending some time thinking about it, I find it extremely reassuring. Ten thousand hours is a long time. But that’s ten thousand hours where, no, you shouldn’t feel like a master. It’s okay to be in the dark. It’s okay to struggle and search and flail. Don’t expect yourself to be a master, because you aren’t one. Be a student, always, and be ready to face the struggles of learning and not knowing things. Don’t let it scare you. And, most importantly, it’s not about having ‘the gift,’ but instead putting in the hours and the struggle to reach mastery.

Learning can be hard. It can be terrifying. It can be demoralizing. But it can also be wonderful, inspiring, invigorating, ecstatic. Setting your sights on a stretch project and making it actually happening is sublime. Learning a new way to handle a problem that’s been plaguing you feels better than a fresh shave. Don’t fear learning, and don’t fear the topic you’re studying. Embrace it, aim high, struggle, fail, and learn.

At least, until I went back to school for coding.

While this installation of Seeing Sound doesn’t have any big, flashy set pieces to show off like the last one, I’d like to walk through some of the revelations I’ve had chugging through this project, in hopes that some other souls will glean something of value out of it. Numbers don’t have to be scary, and with a little bit of attention and foresight, can actually be made incredibly friendly in your search to make strange and wonderful software.

My stretch goal for my last project was to create a processing sketch that could handle incoming MIDI data from Ableton Live, and turn it into glorious, live-loopable visuals for my LED matrix. After much finagling and banging of my head against the desk, I’m proud to say I’ve done that… mostly. By using Apple’s built-in IAC driver to create a virtual MIDI bus, and using the Processing midibus library, I was able to get communication set up between the two programs fairly easily. The job didn’t stop there, however. Once I had data on what MIDI notes were being pressed or played, how could I turn those into visuals?

MIDI is the computer’s version of musical notation. Instead of a staff and scales, pianissimo and fortissimo, crescendos and codas, we have a series of possible notes, 0 through 127, each carrying its own velocity (how hard the note was hit) depending on how it was played. On top of that, we have CC (Continuous Control) messages, for all those beautiful knobs and faders, which will send their own values when moved, also from 0 to 127. Various software, from synth plugins to samplers to drum machines and more, can take this MIDI information, and make it feel like you’re actually sitting in front of a grand piano in a concert hall in Stockholm, rather than in a stuffy apartment in front of a goofy looking MIDI keyboard with enough flashing lights on it to read by.

Processing isn’t concerned with what these numbers mean, it just hands you the numbers to do with what you will. So there I was, once again confronted with a barrage of meaningless digits, and no easy way to turn them into the visuals I want.

My first approach was to get something on the page that did… anything. To get the data from midibus to the rest of the sketch, I initialized an array of 127 integers, and dropped a piece of code in my sketch to write the velocity of an incoming note to its corresponding spot in the array.

Now, hitting low C (MIDI note 0) at a velocity of 100 would set note[0] to 100. Great. In my draw() block, I set up a for loop to iterate through my note array, and draw an ellipse for each one, with the transparency of that ellipse set to the value of that note in the array. As a first draft, it worked — hitting a MIDI note in Ableton, with MIDI out routed to our IAC bus, would cause a circle to appear on the screen. Adding another piece of code to the top of our for loop to reduce every value in the array by a set decay rate until it reaches 0 made it feel a little more functional — now, a note press will reset the transparency to whatever the velocity of the press was, and then it will fade back to invisible at a speed determined by our decay rate.

Before I knew it, I had a bulky, ugly sketch in which I was multiplying note[i] by all sorts of weird numbers to get the results somewhere usable. RGBA color definitions operate in ranges of 0 to 255 — it wouldn’t be hard to set up an equation to get our 0–127 value to scale from 0–255 — but we might also want to use the note number to change the location of our circle across our scene, or to change our ellipse’s circle. Suddenly we’re writing dozens of ugly equations all over the sketch. There must be a better way.

Enter map(). The map function allows us to pass in a variable, then that variable’s range, and then a range that we’d like to scale it to. Good: We can use this to replace our transparency equation. Better: We can use this to replace every modification equation on our code. Best: We can map each note once, before it gets stored in the variable, and make it easily applicable to any value in our sketch.

How, you say? Limit the range to 0–1. While I’m sure this is common practice to many, the idea didn’t occur to me until I was working with the Math.random() function in Javascript. By multiplying the output, a random number between 0 and 1, by any number, you could easily set up the random number range to whatever you wanted. Similarly, if we map() our velocity value to 0–1, all we have to do to set transparency perfectly is drop in note[i] * 255.

On the micro level, this made my Processing sketch infinitely easier. On the macro level, I realized that with just a bit of foresight about how you handle data, and what it will be applied to, you can clean up miles of code. An inch of prevention is worth a pound of cure. Now, these numbers suddenly don’t seem so scary, and my code is infinitely more flexible.

You can dig through the current Processing sketch for my LED matrix, including MIDI control, here.

To date, though, I’ve only managed to play in tiny bars with no more than a couple gelled source-4’s pointed at seemingly random directions around the stage. Not exactly the vibrant audiovisual barrage I imagined.

But everything changed when I learned how awesome Processing could be.

After my last blog post, I started eyeing the corpse of an old project — an LED matrix made with Neopixels that I’d strapped to the ceiling in my room and used as overly elaborate mood lighting. It was an experiment I took on way before I had any idea what I was doing with code, and I stumbled through the Processing side of things by chopping together bits of other peoples’ code, and had it running for a grand total of a week before I melted the LED strips by running it off a power supply with way too much voltage. Since then, the 5-foot strips of LED haven’t been good for much more than my cat’s entertainment.

Suddenly, with a newfound handle on Processing and a lesson well learned about how electricity works, it seemed like a great time for a more functional encore of the LED matrix.

Neopixels are a brand-name version of WS2812B LED strips, extremely bright and colorful little buggers that can each be individually addressed by a controller — meaning unlike a lot of LED mood-lighting strips, each one can be told to display a different color, like a TV screen or computer monitor. To drive them, I’ve picked up a Fadecandy, a tiny microcontroller (limited mini computer) designed specifically to drive LEDs. It handles the bulk of the image processing and layers on some extra icing like smoothing out fades — LEDs don’t like fading smoothly to black — and making colors pop. Awesome as the Fadecandy is, though, it has a one-track mind, and it doesn’t have room to think about receiving images, just talking to the LEDs.

For the heavy lifting, I’ve plugged the Fadecandy into a Raspberry Pi, a wonderful little customizable credit card sized computer. Once I got it up and running, I installed the Fadecandy Server, prepped the Pi for communication over a network, and told it to run a Fadecandy Server whenever it boots. Ideally, this means I’ll never have to hook it up to a monitor and keyboard again, and I can handle tweaks and edits from any other computer on my network.

Next came several hours of soldering. Each of the Fadecandy’s 8 channels can handle up to 64 LEDs. I chose to connect two strips of 30 to each of four channels, for a total of 8 rows of 30 pixels each. Part of the Fadecandy’s magic is that you can tell it, via FCServer’s settings and a Processing library, how your matrix is arranged, so even though each channel is 60 pixels split in two running up and back down the mounting board, it can correctly address each pixel’s location. Finally, I mounted all eight strips on a 4' plank of plywood, leaving some room on the far side to mount the Fadecandy, Raspberry Pi, and some power busses to fuel it all.

Finally, it was time for the fun stuff. OPC (Open Pixel Control) is a library for Processing that handles orienting the pixels and transmitting — wirelessly! — to the Pi. After working out the block of code to define where each strip should live on the screen, I had a portable initializer for my LED matrix that I could drop into any sketch I wanted. To test, I fired up my visualizer sketch from the last blog — gloriously, impossibly, the board lit up like a Bon Jovi concert. My tiny Brooklyn apartment could barely handle the amount of illumination spilling out of it. I like to imagine that it looked something like the stereotypical mad scientist’s laboratory, with random flashes of light spilling out from the windows and underneath the door.

As cool as the visualizer looked, though, the finer details and small rings didn’t translate well to the extremely low-resolution screen I’d created. Additionally, it wasn’t very flexible. I needed to take it further, make it practical and modular, somehow. After some furious googling, I came across Syphon, a lovely piece of software that routes video between applications on your Mac, much like Loopback, the audio-routing software I’ve been using for the visualizer. After loading in the Syphon library, I could stream video from Resolume, a professional VJ (video jockey) application I picked up during my misguided attempts to do concert visuals in college.

Syphon turned out to be remarkably easy to work with. The Processing code required is pretty basic — the only stumbling block was matching framerates. Over time, the framerate of Resolume would desynchronize with the refresh rate of Processing, leading to some extreme flickering that looked pretty ugly on the pixels. Telling both Resolume and Processing to operate at 30 frames has largely smoothed that out, though, and now I have a lovely, beat-syncable, wireless, gigantic wall of lights that I can bring with me when I finally start my world tour.

What’s next? The more I use Resolume, the more I realize it’s a limiting factor to how crazy I can get with the lights. It tempo matching is limited at best, and for someone used to music sequencers, which are extremely flexible and well-integrated, it just couldn’t get me where I wanted. Luckily, I already have a fantastic piece of software that can send detailed signals digitally — Ableton Live. Since Processing can take MIDI signals, I should be able to tell it to listen for MIDI routed out of Ableton, and then use Ableton’s incredibly fluid sequencing and looping functions to create complex visuals on the fly. The catch is I’m going to have to make a monster of a Processing sketch to get all the functions I see in my head, out onto the matrix.

With any luck, my next blog post will be the glorious and flashy conclusion of this three-part journey into the wonderful world of Processing.

So, naturally, as soon as I had the barest grasp of some general programming concepts, I started thinking about how I could use my newfound skills to mess with noise. My searches turned up Processing, a language designed around art. It can be used to animate, to create audiovisual experiences, and even to create sound. But for today’s experiment, we’re going to use it to see what our sound is doing.

Our weapon of choice in this endeavor is the Fast Fourier Transform, or FFT. The FFT will allow us to break an incoming audio signal into separate bands, each spanning only a tiny frequency spectrum, and analyze the amount of signal, or amplitude, in realtime. We’ll accomplish this through Minim, a Processing library. You can download this from within Processing itself, by going to Sketch -> Import Library -> Add Library in the menu bar. Search for Minim and install it.

Now that we have Minim running, we can get started with the code. First, we need to import and start Minim, prepare our audio input, and get ready for our FFT. In Processing, our setup() block is for functions that need to be prepared before animation, and our draw() block is code that gets processed every frame to create movement and animation.

Once we have the prep work complete, we can start our void draw() block. Remember, this block repeats every frame of animation — and Processing isn’t particularly smart on its own— so although we could have prepared parameters like background color in our setup, we’ll enter it at the top of Draw instead so that our window refreshes itself every frame.

Next, fft.forward(in.mix); will forward our input buffer, and the stroke() command is defining our line color in RGBA. Feel free to get creative. The for loop is where the action happens, though. This loop will iterate through every band of our FFT analysis and draw a line based on its height. The first two arguments of line() are our starting coordinates x and y, and the second two our ending coordinates.

And voila. If we run our code now, we’ll get a realtime FFT diagram of our mic input. While it might not be the most exciting piece of code, this is the foundation of any audio reactive code you could hope to create. By iterating through the different bands of our FFT analysis, we can check the amount of content at any frequency range — so if you want the background to change color in relation to the bass, you can do that. If you want sparks of light to appear with the high-frequency cymbals, you can do that. So grab a coffee, put on some good music, and experiment!