How to render synchronous audio and video in Processing in 2021

About a decade ago I wrote a blog post about rendering synchronous audio and video in processing. I posted it on my now-defunct blog, computermusicblog.com. Recently, I searched for the same topic, and found that my old post was one of the top hits, but my old blog was gone.

So in this post I want to give searchers an updated guide for rendering synchronous audio and video in processing.

It’s still a headache to render synchronous audio and video in Processing, but with the technique here you should be able to copy my work and create a simple 2-click process that will get you the results you want in under 100 lines of code.

Prerequisites

You must install Processing, Minim, VideoExport, and ffmpeg on your computer. Processing can be installed from processing.org/. Minim and VideoExport are Processing libraries that you can add via Processing menus (Sketch > Import Library > Add Library). You must add ffmpeg to your path. Google how to do that.

The final, crappy prerequisite for this particular tutorial is that you must be working with a pre-rendered wav file. In other words, this will work for generating Processing visuals that are based on an audio file, but not for Processing sketches that synthesize video and audio at the same time.

Overview

Here’s what the overall process looks like.

  1. Run the Processing sketch. Press q to quit and render the video file.
  2. Run ffmpeg to combine the source audio file with the rendered video.

Source Code

Without further ado, here’s the source. This code is a simple audio visualizer that paints the waveform over a background image. Notice the ffmpeg instructions in the long comment at the top.

/*
  This is a basic audio visualizer created using Processing.
  
  Press q to quit and render the video.
  
  For more information about Minim, see http://code.compartmental.net/tools/minim/quickstart/
  
  For more information about VideoExport, see https://timrodenbroeker.de/processing-tutorial-video-export/

  Use ffmpeg to combine the source audio with the rendered video.
  See https://superuser.com/questions/277642/how-to-merge-audio-and-video-file-in-ffmpeg
  
  The command will look something like this:
  ffmpeg -i render.mp4 -i data/audio.wav -c:v copy -c:a aac -shortest output.mp4
  
  I prefer to add ffmpeg to my path (google how to do this), then put the above command
  into a batch file.
*/

// Minim for playing audio files
import ddf.minim.*;

// VideoExport for rendering videos
import com.hamoid.*;

// Audio related objects
Minim minim;
AudioPlayer song;
String audioFile = "audio.wav"; // The filename for your music. Must be a 16 bit wav file. Use Audacity to convert.

// image related objects
float scaleFactor = 0.25f; // Multiplied by the image size to set the canvas size. Changing this is how you change the resolution of the sketch.
int middleY = 0; // this will be overridden in setup
PImage background; // the background image
String imageFile = "background.jpg"; // The filename for your background image. The file must be present in the data folder for your sketch.

// video related objects
int frameRate = 24; // This framerate MUST be achievable by your computer. Consider lowering the resolution.
VideoExport videoExport;

public void settings() {
  background = loadImage(imageFile);
  
  // set the size of the canvas window based on the loaded image
  size((int)(background.width * scaleFactor), (int)(background.height * scaleFactor));
}

void setup() {
  frameRate(frameRate);
  
  videoExport = new VideoExport(this, "render.mp4");
  videoExport.setFrameRate(frameRate);
  videoExport.startMovie();
  
  minim = new Minim(this);
  
  // the second param sets the buffer size to the width of the canvas
  song = minim.loadFile(audioFile, width);
  
  middleY = height / 2;
  
  if(song != null) {
    song.play();
  }

  fill(255);
  stroke(255);
  strokeWeight(2);
  
  // tell Processing to draw images semi-transparent
  tint(255, 255, 255, 80);
}

void draw() {
  image(background, 0, 0, width, height);
  
  for(int i = 0; i < song.bufferSize() - 1; i++) {
    line(i, middleY + (song.mix.get(i) * middleY), i+1, middleY + (song.mix.get(i+1) * middleY));
  }
  
  videoExport.saveFrame(); // render a video frame
}

void keyPressed() {
  if (key == 'q') {
    videoExport.endMovie(); // Render a silent mp4 video.
    exit();
  }
}

Drowning in 2020

What a year.

I remember the year we had our second child, Leta. That was a difficult year. Having a two year old and a newborn in the house was not easy. Ultimately, I failed that test in many ways. I don’t think I failed as a father, but I did a lot of damage to myself in order to get through the year.

I changed jobs shortly after Leta was born, but I was never able to succeed at that job. The anxiety from the birth and changing jobs just took over my life. I developed ulcers and other health issues that took me years to beat.

This year was harder. I know I’m not alone when I say that I felt like I was drowning. With the virus ravaging the country and a president who insisted on calling it a hoax or downplaying its effects, it felt like the world was closing in on me.

And that’s what produced this album. I wrote it in the first few months of lockdown, when there was much false hope, but no real hope for an end to the unfolding tragedy.

This music differs from most of my other music in many ways. For one, there’s only one collaborator on it, unlike my other recent work which has featured many other collaborators. Also, it’s much darker. It’s a bleak, synthesized hellscape that chokes off the light. It’s violent and dark and lonely.

Just like 2020.

I’m proud to say that I didn’t fail this year. As the world was melting down around me, I didn’t drown. I swam. I was promoted at my job. I took a leadership position in my community. I volunteered at the local library. I made things. And I didn’t come out with any new health problems.

But still, the year mostly felt like I was drowning.

Thanks for listening!

FYNIX is Drowning in December

I think it’s time for a new collection of solo music from my pop/EDM/hiphop alias, FYNIX. The new EP is called FYNIX is Drowning, and it’s all music that I’ve written during 2020. As you can imagine, it’s a little darker than my usual stuff. I just released the first track on SoundCloud.

We all get ghosted sometimes

Collaboration is a struggle. Often, even when someone says that they want to work together, they may not really know what they want. Even if they do produce something, you may not like it, or they may not like what you do.

This remix is one such example. I saw a Reddit post seeking remixes on a nice original electronic track by an artist named Informal. I downloaded the stems and played some keyboards under them. Then I sent the mix to my buddy Drizztopher Walken and he laid down some vocals.

I sent it back to the original artist and… nothing. Ghosted.

Oh well. It happens more often than I’d like to admit. And if I’m honest, I sometimes have to do it with other people too. I’ve never ghosted anyone, but I have politely told them that our track wasn’t working out. I had to tell them that I had too much going on to keep investing in a collaboration that I knew wasn’t going to work out for me.

It sucks to be on both ends of that transaction, but as an avid collaborator who has worked with hundreds of artists, I know that it happens. I still like the track that we produced, even if the original artist didn’t like it.

Easy Transcriptions of Frozen II Songs Into the Unknown and Some Things Never Change

As I write this, the official sheet music for Frozen II is not yet available, but my daughters want to sing songs from Frozen II in their upcoming recital. So I transcribed their choices into a format that is easy for a single vocalist, and easy for me to play. I am sharing them here just in case this helps anyone else.

Notes

  • I did not transcribe the melodies. I play those by ear where necessary.
  • I removed key changes.
  • I transposed Into the Unknown down 3 half-steps to C-minor. It’s still very high.
  • I removed the interludes for other characters and for the chorus.
  • I don’t know if I got the modulation-heavy outro to Into the Unknown quite right in a different key, but it’s pretty close.

Into the Unknown: Frozen2_IntoTheUnknown

Some Things Never Change: Frozen2_SomeThingsNeverChange