Evan X. Merz

About a decade ago I wrote a blog post about rendering synchronous audio and video in processing. I posted it on my now-defunct blog, computermusicblog.com. Recently, I searched for the same topic, and found that my old post was one of the top hits, but my old blog was gone.

So in this post I want to give searchers an updated guide for rendering synchronous audio and video in processing.

It's still a headache to render synchronous audio and video in Processing, but with the technique here you should be able to copy my work and create a simple 2-click process that will get you the results you want in under 100 lines of code.

Prerequisites

You must install Processing, Minim, VideoExport, and ffmpeg on your computer. Processing can be installed from processing.org/. Minim and VideoExport are Processing libraries that you can add via Processing menus (Sketch > Import Library > Add Library). You must add ffmpeg to your path. Google how to do that.

The final, crappy prerequisite for this particular tutorial is that you must be working with a pre-rendered wav file. In other words, this will work for generating Processing visuals that are based on an audio file, but not for Processing sketches that synthesize video and audio at the same time.

Overview

Here's what the overall process looks like.

1. Run the Processing sketch. Press q to quit and render the video file.
2. Run ffmpeg to combine the source audio file with the rendered video.

Source Code

Without further ado, here's the source. This code is a simple audio visualizer that paints the waveform over a background image. Notice the ffmpeg instructions in the long comment at the top.

/*
  This is a basic audio visualizer created using Processing.
  
  Press q to quit and render the video.
  
  For more information about Minim, see http://code.compartmental.net/tools/minim/quickstart/
  
  For more information about VideoExport, see https://timrodenbroeker.de/processing-tutorial-video-export/

  Use ffmpeg to combine the source audio with the rendered video.
  See https://superuser.com/questions/277642/how-to-merge-audio-and-video-file-in-ffmpeg
  
  The command will look something like this:
  ffmpeg -i render.mp4 -i data/audio.wav -c:v copy -c:a aac -shortest output.mp4
  
  I prefer to add ffmpeg to my path (google how to do this), then put the above command
  into a batch file.
*/

// Minim for playing audio files
import ddf.minim.*;

// VideoExport for rendering videos
import com.hamoid.*;

// Audio related objects
Minim minim;
AudioPlayer song;
String audioFile = "audio.wav"; // The filename for your music. Must be a 16 bit wav file. Use Audacity to convert.

// image related objects
float scaleFactor = 0.25f; // Multiplied by the image size to set the canvas size. Changing this is how you change the resolution of the sketch.
int middleY = 0; // this will be overridden in setup
PImage background; // the background image
String imageFile = "background.jpg"; // The filename for your background image. The file must be present in the data folder for your sketch.

// video related objects
int frameRate = 24; // This framerate MUST be achievable by your computer. Consider lowering the resolution.
VideoExport videoExport;

public void settings() {
  background = loadImage(imageFile);
  
  // set the size of the canvas window based on the loaded image
  size((int)(background.width * scaleFactor), (int)(background.height * scaleFactor));
}

void setup() {
  frameRate(frameRate);
  
  videoExport = new VideoExport(this, "render.mp4");
  videoExport.setFrameRate(frameRate);
  videoExport.startMovie();
  
  minim = new Minim(this);
  
  // the second param sets the buffer size to the width of the canvas
  song = minim.loadFile(audioFile, width);
  
  middleY = height / 2;
  
  if(song != null) {
    song.play();
  }

  fill(255);
  stroke(255);
  strokeWeight(2);
  
  // tell Processing to draw images semi-transparent
  tint(255, 255, 255, 80);
}

void draw() {
  image(background, 0, 0, width, height);
  
  for(int i = 0; i < song.bufferSize() - 1; i++) {
    line(i, middleY + (song.mix.get(i) * middleY), i+1, middleY + (song.mix.get(i+1) * middleY));
  }
  
  videoExport.saveFrame(); // render a video frame
}

void keyPressed() {
  if (key == 'q') {
    videoExport.endMovie(); // Render a silent mp4 video.
    exit();
  }
}

This summer I'm going to release a new album of solo electronic music that is heavily influenced by EDM and classic rock. For the past few weeks I've been trying to figure out what to do about the art on the new album.

The album is called "FYNIX Fights Back" so I decided to use images of animals fighting, but I didn't just want to pull up generic images. I wanted to add my own special touch to each image. So I pulled out a tool that I haven't used in years: the Processing programming language.

Processing is great for simple algorithmic art. In this case I wanted to glitch some images interactively, but I wasn't exactly sure what I wanted to do.

So I just started experimenting. I played with colors and shapes and randomness. I like to derive randomess based on mouse movement. The more the mouse moves, the more randomness is imparted to whatever the user is doing.

I added image glitching based on mouse speed. The quicker the cursor moves, the more random are the generated shapes, and the more they are offset from their source position in the original image.

Here's the end result.

Here's the source code. Make it even better.

import java.awt.event.KeyEvent;

// relationship between image size and canvas size
float scalingFactor = 2.0;

// image from unsplash.com
String imageFilename = "YOUR_FILE_IN_DATA_FOLDER.jpg";

// image container
PImage img;
PImage scaledImage;

int minimumVelocity = 40;
int velocityFactorDivisor = 5; // the larger this is, the more vertices you will get
int velocityFactor = 1; // this will be overridden below

float minimumImageSize = 10.0f;

boolean firstDraw = true;

int currentImage = 1;

void settings() {
  // load the source image
  img = loadImage(imageFilename);
  
  // load the pixel colors into the pixels array
  img.loadPixels();
  
  // create a canvas that is proportional to the selected image
  size((int)(img.width / scalingFactor), (int)(img.height / scalingFactor));
  
  // scale the image for the window size
  scaledImage = loadImage(imageFilename);
  scaledImage.resize(width, height);
  
  // override velocityFactor
  velocityFactor = (int)(width / velocityFactorDivisor);
}

void setup() {
  // disable lines
  noStroke();
}

void keyPressed() {
  if(keyCode == KeyEvent.VK_R) {
    firstDraw = true; // if the user presses ENTER, then reset
  }
}

void draw() {
  if(firstDraw) {
    image(scaledImage, 0, 0);
    firstDraw = false;
  }
  
  // right click to render to image file
  if(mousePressed && mouseButton == RIGHT) {
    save(imageFilename.replace(".jpg", "") + "_render_" + currentImage + ".tga");
    currentImage++;
  }
  
  if(mousePressed && mouseButton == LEFT && mouseX >= 0 && mouseX < width && mouseY >= 0 && mouseY < height) {
    int velocityX = minimumVelocity + (3 * velocity(mouseX, pmouseX, width));
    int velocityY = minimumVelocity + (3 * velocity(mouseY, pmouseY, height));
    
    color c = img.pixels[mousePositionToPixelCoordinate(mouseX, mouseY)];

    int vertexCount = ((3 * velocityFactor) + velocityX + velocityY) / velocityFactor;
    
    int minimumX = mouseX - (velocityX / 2);
    int maximumX = mouseX + (velocityX / 2);
    int minimumY = mouseY - (velocityY / 2);
    int maximumY = mouseY + (velocityY / 2);
    
    PGraphics pg = createGraphics(maximumX - minimumX, maximumY - minimumY);
    
    pg.beginDraw();
    pg.noStroke();
    pg.fill(c);

    // first draw a shape into the buffer
    pg.beginShape();
    for(int i = 0; i < vertexCount; i++) {
      pg.vertex(random(0, pg.width), random(0, pg.height));
    }
    pg.endShape();
    pg.endDraw();
    
    pg.loadPixels();
    
    // then copy image pixels into the shape
    
    // get the upper left coordinate in the source image
    int startingCoordinateInSourceImage = mousePositionToPixelCoordinate(minimumX, minimumY);
    
    // get the width of the source image
    int sourceImageWidth = (int)(img.width);
    
    // set the offset from the source image
    int offsetX = velocity(mouseX, pmouseX, width);
    int offsetY = velocity(mouseY, pmouseY, height);
    
    // ensure that the offset doesn't go off the canvas
    if(mouseX > width / 2) offsetX *= -1;
    if(mouseY > height / 2) offsetY *= -1;
    
    for(int y = 0; y < pg.height; y++) {
      for(int x = 0; x < pg.width; x++) {
        // calculate the coordinate in the destination image
        int newImageY = y * pg.width;
        int newImageX = x;
        int newImageCoordinate = newImageX + newImageY;
        
        // calculate the location in the source image
        //int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + (x * scalingFactor) + ((y * scalingFactor) * sourceImageWidth));
        //int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + ((x + offsetX) * scalingFactor) + (((y + offsetY) * scalingFactor) * sourceImageWidth));

        int sourceImageX = (int)(((x + offsetX) * scalingFactor));
        int sourceImageY = (int)((((y + offsetY) * scalingFactor) * sourceImageWidth));
        int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + sourceImageX + sourceImageY);
        
        // ensure the calculated coordinates are within bounds
        if(newImageCoordinate > 0 && newImageCoordinate < pg.pixels.length 
           && sourceImageCoordinate > 0 && sourceImageCoordinate < img.pixels.length
           && pg.pixels[newImageCoordinate] == c) {
          pg.pixels[newImageCoordinate] = img.pixels[sourceImageCoordinate];
        }
      }
    }
    
    image(pg, minimumX, minimumY);
  }
}

// convert the mouse position to a coordinate within the source image
int mousePositionToPixelCoordinate(int mouseX, int mouseY) {
  return (int)(mouseX * scalingFactor) + (int)((mouseY * scalingFactor) * img.width);
}

// This sort of calculates mouse velocity as relative to canvas size
int velocity(float x1, float x2, float size) {
  int val = (int)(Math.abs((x1 - x2) / size) * size);
  return val;
}

Quincy Tahoma painted The Last Jump in 1954. The subject matter and execution issues reflect the problems in Tahoma's life.

The scene should be familiar to any Tahoma fan. The painting shares a title with a famous Tahoma print. In fact this scene is one he painted frequently because it was easy to sell. A man is trying to break a horse, but the horse is startled by a small animal, a bunny in this case, and the man is thrown by the horse. We can see that he is thrown in the cartouche.

The painting has some of the hallmarks of Tahoma's best works. It depicts the most dramatic moment of the effort, where the story could end in success or failure. The horse is nearly flawlessly executed. The poses of the horse and the man are extremely three dimensional, with limbs jutting out at the viewer, and flailing in dramatic ways. The drama is what separates Tahoma's work from his Navajo contemporaries, who often favored flat or static scenes.

Yet when you spend a little more time looking at this painting, you can see that something isn't quite right. It looks like the paintbrush slipped when painting the horse's left nostril and there's a black splotch right next to it. The torso of the man is unfinished, with the outline missing from his right side, and his abs hastily outlined with a liner brush. The green of the man's loin cloth is flat green, as opposed to the bright, multifaceted green that he almost always used for the same piece of clothing in his other paintings.

What happened?

Quincy Tahoma died in 1956 due to complications of alcoholism. In 1954 he drank and painted every day. His work from this time is quite inconsistent. These inconsistencies mar this image. When I look at this image I see a painting that was started in the morning, when he was sober, then hastily finished in the evening when he was drunk. The pose, the horse and the headdress are almost flawlessly planned and executed. This headdress is one of the most spectacular I've ever seen in a Tahoma painting, and the horse reveals Tahoma's skill at capturing the animals.

I imagine Tahoma taking a break after painting most of the picture, continuing to drink, then returning to finish it. He hastily filled in the loin cloth with a basic green straight from the bottle. Then he dabbed some black into the horse's nostrils, but his arm slipped and he splotched it. Then he got angry and decided to do the signature. His arm slipped when making the top of the 5 and he dragged it across where the 4 would go. He took his time and finished the cartouche strong, but then he noticed that he forgot to finish the man's torso. Using the same narrow brush he used for the cartouche, he quickly outlined the man's muscles. Even then, he still forgot one small inner line.

Of course this narrative is pure fancy, but the story told by this painting is plain: the man failed to tame the horse just as Tahoma failed to tame his alcoholism.

Quincy Tahoma painted Over the Hollow Log in 1947. This painting shows an artist at the peak of his skills. Everything about this image is characteristic of the best Tahoma paintings. But why was he doing his best work in 1947?

In 1947 Tahoma was 30 years old. He was a famous artist in New Mexico, and he had achieved a measure of national success by winning a prize from the Philbrook Museum in 1946. But he was facing the first great challenge of his life as a consequence of his growing dependence on alcohol.

The reason Tahoma painted some of his best pieces in 1947 is because he was sober. The reason he was sober is that he was in prison. The reason he was in prison is because he was accused of rape, and the story reveals something about Tahoma's character, and about how the criminal justice system treated Native Americans in the mid 20th century.

Here's how Charnell Havens and Vera Marie Badertscher summarized the incident that landed Tahoma in prison in 1947 in Quincy Tahoma: The Life and Legacy of a Navajo Artist.

On New Year's Eve 1946, he had many successes to celebrate. He liked to socialize and his celebrations often involved too much drinking. He would sober up – sometimes in jail – and then get back to life. This time, however, the party got out of control…

On New Year's Day, a Wednesday, Mrs. Hobbs complained to the police that her daughter was missing. When the sheriff, aided by the state patrol, located the girl in Albequerque on that same day, she "told her story" according to the Santa Fe newspaper. She said a Navajo artist named Quincy Tahoma had offered her whiskey and raped her. The police quickly located Tahoma and held him in the Santa Fe jail until court convened on Friday, January 3…

Quincy Tahoma stood alone in the courtroom, probably not feeling well after the New Year's Eve celebration, and someone in the court or the District Attorney's office advised that he could plead guilty to a lesser plea. No defense attorney appears in the court paperwork so it is doubtful that anyone explained his options to him. No one suggested that if he thought the charges were unfair, he could plead not guilty. No transcript of witnesses appears, and the trial did not take long.

Tahoma was accused of raping a 16 year old girl named Vonnie Hobbs who had been out with him on New Year's Eve. He pled guilty and received a sentence of two to five years. His signature appears on none of the documents related to the case.

But prison was not so bad for Tahoma. In fact, if we judge that period of his life based on his artwork, it was the best period of his life. Tahoma was a famous prisoner, and he was well known to the warden and the guards. The guards purchased supplies for him, and he was able to paint every day. Since painting was all he cared about, he was not much bothered by being in prison.

He gave some paintings to the warden and the guards for additional privileges. Other paintings were taken out of prison and sold by various friends. Over the Hollow Log is almost certainly a painting that Tahoma produced while in prison.

Tahoma's work outside prison was becoming quite inconsistent. His drinking was lending some paintings an unfinished or sketchy quality. None of his prison work reflects this drop in quality.

The painting is one of Tahoma's best because it reflects his mastery of single page storytelling. It depicts a Native American man wearing a concho belt shooting a second arrow into a buck, while his horse and the buck leap a rotting log. The deer breathes its last breath while a scared rabbit runs out of the log. In the cartouche we can see that the hunter has managed to bring down the deer, as he has tied his kill to the back of his mount.

So much of this image is Quincy Tahoma flexing the techniques he had mastered by 1947. The background is plain, unpainted paper, but the scene is framed by three birds in the sky. The composition and drama of the scene reflects elements that were palatable to him, and were easy to sell to tourists. The horse and deer are painted using a mixture of painting and inking techniques that are still so evocative today. The image almost looks like an illustration from a comic book or pulp novel. The small animal running away appears in so many Tahoma paintings. One of his most oft repeated subjects was a horse being scared by a skunk or other small animal. The dress of the hunter is also typical of Tahoma, with the characteristic flourescent blue-green loin cloth that appears in almost every Tahoma depiction of Native Americans in traditional dress.

The depiction of the tree in this painting is remarkably life-like. Tahoma preferred to paint stylized plants and trees. His depiction of a realistic tree in this painting is one thing that makes it stand out among his other works.

Tahoma lived many lives. After the war he had an increasingly difficult time keeping his lives separate. In 1947 his double lives continued. He was simultaneously a prisoner whose basic freedoms had been taken away, and a successful artist creating the best work of his life.

Painted during his time in prison, Over the Hollow Log is one of the best pieces created by Quincy Tahoma.

I have so much to say about 2020 that I'm not quite sure how to say it. The year was simultaneously fantastic and terrible for me and my family. A lot of good things happened to me to 2020, and I was able to avoid a lot of the worst things that were happening around the country.

I'm having a hard time talking about 2020 because, among so much suffering, my family actually did pretty well. Nobody in my family died of COVID. My kids are doing well with remote learning. My wife and I have begun volunteering in our community, and really getting to know our community. I even got a promotion at work.

I want to talk about all of this, but it feels like gloating. The pandemic is not over yet. There are so many people still suffering so much due to the negligence, stupidity, and incompetence of our elected Republican leaders.

So what can I say?

I can say that 2020 changed me. It changed me in a lot of ways.

One of the odd, quirky traits that I developed during lockdown is a passionate interest in 20th century Navajo artists, especially those taught at the Santa Fe Indian School by Dorothy Dunn. This includes people like Harrison Begay, Woody Crumbo, and Allan Houser. But for me, the best among them was Quincy Tahoma.

I don't know how this interest developed, but it's so unusual that it's the one thing I feel comfortable writing about at this point. Which is a long winded way of saying that, although I'm no expert, I'm going to write about some of the art that I've fallen in love with in the past year.