How to render synchronous audio and video in Processing in 2021

About a decade ago I wrote a blog post about rendering synchronous audio and video in processing. I posted it on my now-defunct blog, computermusicblog.com. Recently, I searched for the same topic, and found that my old post was one of the top hits, but my old blog was gone.

So in this post I want to give searchers an updated guide for rendering synchronous audio and video in processing.

It’s still a headache to render synchronous audio and video in Processing, but with the technique here you should be able to copy my work and create a simple 2-click process that will get you the results you want in under 100 lines of code.

Prerequisites

You must install Processing, Minim, VideoExport, and ffmpeg on your computer. Processing can be installed from processing.org/. Minim and VideoExport are Processing libraries that you can add via Processing menus (Sketch > Import Library > Add Library). You must add ffmpeg to your path. Google how to do that.

The final, crappy prerequisite for this particular tutorial is that you must be working with a pre-rendered wav file. In other words, this will work for generating Processing visuals that are based on an audio file, but not for Processing sketches that synthesize video and audio at the same time.

Overview

Here’s what the overall process looks like.

  1. Run the Processing sketch. Press q to quit and render the video file.
  2. Run ffmpeg to combine the source audio file with the rendered video.

Source Code

Without further ado, here’s the source. This code is a simple audio visualizer that paints the waveform over a background image. Notice the ffmpeg instructions in the long comment at the top.

/*
  This is a basic audio visualizer created using Processing.
  
  Press q to quit and render the video.
  
  For more information about Minim, see http://code.compartmental.net/tools/minim/quickstart/
  
  For more information about VideoExport, see https://timrodenbroeker.de/processing-tutorial-video-export/

  Use ffmpeg to combine the source audio with the rendered video.
  See https://superuser.com/questions/277642/how-to-merge-audio-and-video-file-in-ffmpeg
  
  The command will look something like this:
  ffmpeg -i render.mp4 -i data/audio.wav -c:v copy -c:a aac -shortest output.mp4
  
  I prefer to add ffmpeg to my path (google how to do this), then put the above command
  into a batch file.
*/

// Minim for playing audio files
import ddf.minim.*;

// VideoExport for rendering videos
import com.hamoid.*;

// Audio related objects
Minim minim;
AudioPlayer song;
String audioFile = "audio.wav"; // The filename for your music. Must be a 16 bit wav file. Use Audacity to convert.

// image related objects
float scaleFactor = 0.25f; // Multiplied by the image size to set the canvas size. Changing this is how you change the resolution of the sketch.
int middleY = 0; // this will be overridden in setup
PImage background; // the background image
String imageFile = "background.jpg"; // The filename for your background image. The file must be present in the data folder for your sketch.

// video related objects
int frameRate = 24; // This framerate MUST be achievable by your computer. Consider lowering the resolution.
VideoExport videoExport;

public void settings() {
  background = loadImage(imageFile);
  
  // set the size of the canvas window based on the loaded image
  size((int)(background.width * scaleFactor), (int)(background.height * scaleFactor));
}

void setup() {
  frameRate(frameRate);
  
  videoExport = new VideoExport(this, "render.mp4");
  videoExport.setFrameRate(frameRate);
  videoExport.startMovie();
  
  minim = new Minim(this);
  
  // the second param sets the buffer size to the width of the canvas
  song = minim.loadFile(audioFile, width);
  
  middleY = height / 2;
  
  if(song != null) {
    song.play();
  }

  fill(255);
  stroke(255);
  strokeWeight(2);
  
  // tell Processing to draw images semi-transparent
  tint(255, 255, 255, 80);
}

void draw() {
  image(background, 0, 0, width, height);
  
  for(int i = 0; i < song.bufferSize() - 1; i++) {
    line(i, middleY + (song.mix.get(i) * middleY), i+1, middleY + (song.mix.get(i+1) * middleY));
  }
  
  videoExport.saveFrame(); // render a video frame
}

void keyPressed() {
  if (key == 'q') {
    videoExport.endMovie(); // Render a silent mp4 video.
    exit();
  }
}

Glitching Images in Processing

This summer I’m going to release a new album of solo electronic music that is heavily influenced by EDM and classic rock. For the past few weeks I’ve been trying to figure out what to do about the art on the new album.

The album is called “FYNIX Fights Back” so I decided to use images of animals fighting, but I didn’t just want to pull up generic images. I wanted to add my own special touch to each image. So I pulled out a tool that I haven’t used in years: the Processing programming language.

Processing is great for simple algorithmic art. In this case I wanted to glitch some images interactively, but I wasn’t exactly sure what I wanted to do.

So I just started experimenting. I played with colors and shapes and randomness. I like to derive randomess based on mouse movement. The more the mouse moves, the more randomness is imparted to whatever the user is doing.

I added image glitching based on mouse speed. The quicker the cursor moves, the more random are the generated shapes, and the more they are offset from their source position in the original image.

Here’s the end result.

FYNIX Fights Back. Coming in Summer 2021.

Here’s the source code. Make it even better.

import java.awt.event.KeyEvent;

// relationship between image size and canvas size
float scalingFactor = 2.0;

// image from unsplash.com
String imageFilename = "YOUR_FILE_IN_DATA_FOLDER.jpg";

// image container
PImage img;
PImage scaledImage;

int minimumVelocity = 40;
int velocityFactorDivisor = 5; // the larger this is, the more vertices you will get
int velocityFactor = 1; // this will be overridden below

float minimumImageSize = 10.0f;

boolean firstDraw = true;

int currentImage = 1;

void settings() {
  // load the source image
  img = loadImage(imageFilename);
  
  // load the pixel colors into the pixels array
  img.loadPixels();
  
  // create a canvas that is proportional to the selected image
  size((int)(img.width / scalingFactor), (int)(img.height / scalingFactor));
  
  // scale the image for the window size
  scaledImage = loadImage(imageFilename);
  scaledImage.resize(width, height);
  
  // override velocityFactor
  velocityFactor = (int)(width / velocityFactorDivisor);
}

void setup() {
  // disable lines
  noStroke();
}

void keyPressed() {
  if(keyCode == KeyEvent.VK_R) {
    firstDraw = true; // if the user presses ENTER, then reset
  }
}

void draw() {
  if(firstDraw) {
    image(scaledImage, 0, 0);
    firstDraw = false;
  }
  
  // right click to render to image file
  if(mousePressed && mouseButton == RIGHT) {
    save(imageFilename.replace(".jpg", "") + "_render_" + currentImage + ".tga");
    currentImage++;
  }
  
  if(mousePressed && mouseButton == LEFT && mouseX >= 0 && mouseX < width && mouseY >= 0 && mouseY < height) {
    int velocityX = minimumVelocity + (3 * velocity(mouseX, pmouseX, width));
    int velocityY = minimumVelocity + (3 * velocity(mouseY, pmouseY, height));
    
    color c = img.pixels[mousePositionToPixelCoordinate(mouseX, mouseY)];

    int vertexCount = ((3 * velocityFactor) + velocityX + velocityY) / velocityFactor;
    
    int minimumX = mouseX - (velocityX / 2);
    int maximumX = mouseX + (velocityX / 2);
    int minimumY = mouseY - (velocityY / 2);
    int maximumY = mouseY + (velocityY / 2);
    
    PGraphics pg = createGraphics(maximumX - minimumX, maximumY - minimumY);
    
    pg.beginDraw();
    pg.noStroke();
    pg.fill(c);

    // first draw a shape into the buffer
    pg.beginShape();
    for(int i = 0; i < vertexCount; i++) {
      pg.vertex(random(0, pg.width), random(0, pg.height));
    }
    pg.endShape();
    pg.endDraw();
    
    pg.loadPixels();
    
    // then copy image pixels into the shape
    
    // get the upper left coordinate in the source image
    int startingCoordinateInSourceImage = mousePositionToPixelCoordinate(minimumX, minimumY);
    
    // get the width of the source image
    int sourceImageWidth = (int)(img.width);
    
    // set the offset from the source image
    int offsetX = velocity(mouseX, pmouseX, width);
    int offsetY = velocity(mouseY, pmouseY, height);
    
    // ensure that the offset doesn't go off the canvas
    if(mouseX > width / 2) offsetX *= -1;
    if(mouseY > height / 2) offsetY *= -1;
    
    for(int y = 0; y < pg.height; y++) {
      for(int x = 0; x < pg.width; x++) {
        // calculate the coordinate in the destination image
        int newImageY = y * pg.width;
        int newImageX = x;
        int newImageCoordinate = newImageX + newImageY;
        
        // calculate the location in the source image
        //int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + (x * scalingFactor) + ((y * scalingFactor) * sourceImageWidth));
        //int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + ((x + offsetX) * scalingFactor) + (((y + offsetY) * scalingFactor) * sourceImageWidth));

        int sourceImageX = (int)(((x + offsetX) * scalingFactor));
        int sourceImageY = (int)((((y + offsetY) * scalingFactor) * sourceImageWidth));
        int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + sourceImageX + sourceImageY);
        
        // ensure the calculated coordinates are within bounds
        if(newImageCoordinate > 0 && newImageCoordinate < pg.pixels.length 
           && sourceImageCoordinate > 0 && sourceImageCoordinate < img.pixels.length
           && pg.pixels[newImageCoordinate] == c) {
          pg.pixels[newImageCoordinate] = img.pixels[sourceImageCoordinate];
        }
      }
    }
    
    image(pg, minimumX, minimumY);
  }
}

// convert the mouse position to a coordinate within the source image
int mousePositionToPixelCoordinate(int mouseX, int mouseY) {
  return (int)(mouseX * scalingFactor) + (int)((mouseY * scalingFactor) * img.width);
}

// This sort of calculates mouse velocity as relative to canvas size
int velocity(float x1, float x2, float size) {
  int val = (int)(Math.abs((x1 - x2) / size) * size);
  return val;
}

Redux Explained in One Simple Sentence

Redux is a very simple tool that is very poorly explained. In this article, I will explain Redux in a single sentence that is sorely missing from the Redux documentation.

Redux is a pattern for modifying state using events.

That’s it. It’s a dedicated event bus for managing application state. Read on for a more detailed explanation, and for a specific explanation of the vocabulary used in Redux.

1. Redux is a design pattern.

The first thing you need to know is that Redux is not really software per se. True, you can download a Redux “library”, but this is just a library of convenience functions and definitions. You could implement a Redux architecture without using Redux software.

In this sense, Redux is not like jQuery, or React, or most of the libraries you get from npm or yarn. When you learn Redux, you aren’t learning one specific thing, but really just a group of concepts. In computer science, we call concepts like this a design pattern. A design pattern is a group of concepts that simplify an archetypal task (a task that re-occurs in software development). Redux is a design pattern for managing data in a javascript application.

2. Redux is an extension of the Observer Pattern.

I think this is the only bullet point needed to explain Redux to experienced developers.

The observer pattern is the abstract definition of what programmers refer to as events and event listeners. The observer pattern defines a situation where one object is listening to events triggered by another object.

In javascript, we commonly use the observer pattern when we wire up onClick events. When you set up an onClick event, you are saying that an object should listen to an HTML element, and respond when it is clicked. In this scenario, we are usually managing the visual state of the page using events. For example, a switch is clicked and it must respond by changing color.

Redux is a dedicated event bus for managing internal application state using events.

So redux is a way of triggering pre-defined events to modify state in pre-determined ways. This allows us to ensure that state isn’t being modified in ad hoc ways and it allows us to examine the history of application state by looking at the sequence of events that led to that state.

3. Actions are event definitions. Reducers are state modifiers.

The new vocabulary used by Redux makes it seem like it’s something new, when it’s really just a slight twist on existing technology and patterns.

Action = Event definition. Like events in any language, events in Redux have a name and a payload. Events are not active like classes or methods. They are (usually) passive structures that can be contained or transmitted in json.

Reducer = Event listener. A reducer changes application state in response to an action. It’s called a Reducer because it literally works like something you would pass to the reduce function.

Dispatch = Trigger event. The dispatch method can be confusing at first, but it is just a way of triggering the event/action. You should think of it like the methods used to trigger exceptions in most languages. After all, exceptions are just another variation on the Observer pattern. So dispatch is like throw in Java or raise in Ruby.

4. Redux manages local state separately from the backend.

Redux is a pattern for managing only the internal state of the application. It does not dictate anything about how you talk to your API, nor does it dictate the way that your internal state should be connected to the backend state.

It’s a common mistake to put the backend code directly into the Redux action or reducer. Although this can seem sensible at times, tightly coupling the local application state with backend interactions is dangerous and often undesirable. For instance, a user may toggle a switch three or four times just to see what it does. Usually you want to update local state immediately when they hit the switch, but you only want to update the backend after they have stopped.

Conclusion

Redux is not complex. It is a dedicated event bus for managing local application state. Actions are events. Reducers modify state. That’s it. I hope that helps.

Here are a few other useful links:

Single File Components are Good

I was reading the Vue.js documentation recently, and I came across a very compelling little passage. The passage was explaining the thinking behind single file components. In a single file component, the template, javascript logic, and perhaps even styles, are all collocated in the same file.

Putting all the parts of a component into a single file may seem odd to some programmers. After all, shouldn’t we be separating logic from templates? Shouldn’t we be separating styles from logic too?

And the answer is that the right thing to do is situation dependent, and this is something that a lot of modern programmers don’t understand. A lot of programmers seem to think that there are rules, and you must follow the rules. They don’t realize that rules should only be followed when they are helping you.

So the core question is this: is separating a single view into several files really helping your development process?

Here’s what the Vue documentation says.

One important thing to note is that separation of concerns is not equal to separation of file types. In modern UI development, we have found that instead of dividing the codebase into three huge layers that interweave with one another, it makes much more sense to divide them into loosely-coupled components and compose them. Inside a component, its template, logic and styles are inherently coupled, and collocating them actually makes the component more cohesive and maintainable.

So the argument from the Vue developers is that keeping all the code related to one view inside one file is actually more maintainable than separating them.

I tentatively agree with this.

Obviously, it’s good to separate certain things out into their own files. Foundational CSS styles and core JS helper methods should have their own place. But for all the code that makes a component unique, I think it’s easiest for developers if that is in one file.

Here are the problems I face with breaking a view into multiple files.

1. What is each file called? Where is it located?

When you separate a view into multiple files, it becomes difficult to find each file. It’s fine to have a standard for naming and locating files, but in any large software project, there are going to be situations where the standard can’t be followed for one reason or another. Maybe the view was created early in development. Maybe the view is shared. Or maybe the view was created by a trainee developer and the mis-naming was missed in the code review. Whatever the cause, file names and locations can be stumbling blocks.

At my workplace, we have hundreds of templates named item.js.hamlbars. That’s not very helpful, but what’s more troublesome is when, for whatever reason, a list item couldn’t be named item.js.hamlbars. In that case, I’m left searching for files with whatever code words I can guess.

2. Where is the bug?

When you separate a view into multiple files, you don’t know where a bug may be coming from. This is especially troublesome if you are in a situation where conditional logic is being put into the templates. Once you have conditional logic in the templates, then you can’t know if the bug is in the javascript or in the template itself. Furthermore, it then takes developer time to figure out where to put the fix.

Single File Components are a Bonus

Putting all the code for a single view into a single file is not only logical, but it also doesn’t violate the separation-of-concerns ethos. While the pieces are all in the same file, they each have a distinct section. So this isn’t the bad old days of web development, where javascript was strewn across a page in any old place. Rather, collocating all the code files for a single view is a way to take away some common development headaches.

Don’t Use Amazon’s Forks of Open Source Software

This is a warning. Amazon’s forks of open source software are very poorly supported. If you switch to them, you will create problems for yourself. Here’s my three examples from personal experience.

Amazon Aurora

Amazon Aurora is attractive because it’s compatible with PostGreSQL and MySQL, and it is ready for sharding out-of-the-box. Don’t get sucked in though. The deploy and patching process is a poorly supported nightmare.

Recently, Amazon told us we had to have a forced upgrade patch on our Aurora instance. We don’t do downtime. Ever. But Amazon doesn’t provide a no-downtime version of the patch, and the alternative was too much of a time investment.

Amazon guaranteed us the deploy would only take 3minutes. So we decided to go forward with a little but of downtime. The first time it tried to patch it failed and took the database down for 10 minutes. Then it sat there for 15 minutes during which we could do nothing. Then it tried to patch again, taking the database down for another 10 minutes. And it failed again.

So at that point we were frantically calling Amazon support to try to stop it, but we couldn’t get through to anyone who was at all helpful. So the database went down a third time, and this time, fortunately, it worked.

All told, we had about an hour of downtime after Amazon guaranteed us only 3 minutes. The support was unhelpful, and didn’t really know how to help.

Amazon Linux

Do not use Amazon Linux. If that article isn’t enough to convince you, then here’s my story.

I like to try new technologies. I like to keep up to date with the latest thing. I use AWS in my every day job. So when Amazon started pushing their own version of Linux, I decided to try it on my personal website.

The major headache this has created for me is that the Lets Encrypt Certbot doesn’t work on Amazon Linux. So I can’t do automatic renewal of my SSL certificate. I have to step through the process manually each month.

But there are many other tools and packages that are incompatible too. Don’t use it.

Amazon’s Android

Amazon’s version of Android is awful for the same reasons. It’s extremely poorly maintained.

I loved the first Kindle device. It was perfect. It reduced waste, and allowed me to carry a library around with me. So I bought the initial devices, then carried on into the Kindle Fire and other tablets. That’s when the trouble began.

There are major bugs and performance issues on Amazon’s version of Android that have gone unaddressed for years.

The bug that caused me to switch to a Samsung tablet is one where, when I clicked on my most recent media item, it wouldn’t actually open my most recent item. So, for instance, if you are watching a show or reading a book, or switching between two things, then you are constantly opening the wrong thing. Because the performance is so poor, this is a problem that often takes minutes to resolve by clicking around and hoping it opens the correct item.

That seems like a critical bug to me. It should be one that gets fixed immediately, right? Well it existed for years.

Don’t Use Amazon’s Forks of Open Source Software

Don’t do it. That one feature they add isn’t worth it. Use the open source alternative.