Tagged "programming"
Wii Play the Drums
Now with nunchuk support, so you can be both a drummer AND a ninja!
MapHub Virtual Instrument
A demonstration of my virtual instrument based on geo-tagged calls from maphub.org
To the Towers and the Satellites
Here's a video of a recent performance of one of my pieces called To the Towers and the Satellites. Music composed by Evan X. Merz. Sculpture and Stage by Sudhu Tewari. Movement by Nuria Bowart and Shira Yaziv.
Buy Transient on Bandcamp
I'm so pleased to announce the release of a new album of experimental electronic music called Transient.
Transient is a set of music that I wrote in the summer I moved from Chicago to San Jose. At the time, I was experimenting with custom algorithmic audio slicing and glitching software, and you can hear this on many of the tracks.
Sonifying Processing
Sonifying Processing shows students and artists how to bring sound into their Processing programs. It takes a hands-on approach, incorporating examples into every topic that is discussed. Each section of the book explains a sound programming concept then demonstrates it in code. The examples build from simple synthesizers in the first few chapters, to more complex sound-manglers as the book progresses. Each step of the way is examined at a level that is simple enough for new learners, and comfortable for more experienced programmers.
Topics covered include Additive Synthesis, Frequency Modulation, Sampling, Granular Synthesis, Filters, Compression, Input/Output, MIDI, Analysis and everything else an artist may need to bring their Processing sketches to life.
"Sonifying Processing is a great introduction to sound art in Processing. It's a valuable reference for multimedia artists." - Beads Creator Oliver Bown
Sonifying Processing is available as a free pdf ebook, in a Kindle edition, or in print from Amazon.com.
Downloads
Sonifying Processing The Beads Tutorial
The Beads Library was created by Oliver Bown, and can be downloaded at http://www.beadsproject.net/.
Press
Sonifying Processing on Peter Kirn's wonderful Create Digital Music
Disconnected, Algorithmic Sound Collages from Web API
I'm pleased to announce the release of Disconnected, an album of algorithmic sound collages generated by pulling sounds from the web.
I prefer to call this album semi-algorithmic because some of the music is purely software-generated, while other pieces are a collaboration between the software and myself. Tracks four and six are purely algorithmic, while the other tracks are a mix of software-generated material and more traditionally composed material.
The software used in the sound collage pieces (1, 3, 4, 6) was inspired by Melissa Schilling's Small World Network Model of Cognitive Insight. Her theory essentially says that moments of cognitive insight, or creativity, occur whenever a connection is made between previously distantly related ideas. In graph theory, these types of connections are called bridges, and they have the effect of bringing entire neighborhoods of ideas closer together.
I applied Schilling's theory to sounds from freesound.org. My software searches for neighborhoods of sounds that are related by aural similarity and stores them in a graph of sounds. These sounds are then connected with more distant sounds via lexical connections from wordnik.com. These lexical connections are bridges, or moments of creativity. This process is detailed in the paper Composing with All Sound Using the FreeSound and Wordnik APIs.
Finally, these sound graphs must be activated to generate sound collages. I used a modified boids algorithm to allow a swarm to move over the sound graph. Sounds were triggered whenever the population on a vertex surpassed a threshold.
Swarm Intelligence in Music in The Signal Culture Cookbook
The Signal Culture Group just published The Signal Culture Cookbook. I contributed a chapter titled "The Mapping Problem", which deals with issues surrounding swarm intelligence in music and the arts. Swarm intelligence is a naturally non-human mode of intelligent behavior, so it presents some unique problems when being applied to the uniquely human behavior of creating art.
The Signal Culture Cookbook is a collection of techniques and creative practices employed by artists working in the field of media arts. Articles include real-time glitch video processing, direct laser animation on film, transforming your drawing into a fake computer, wi-fi mapping, alternative uses for piezo mics, visualizing earthquakes in real time and using swarm algorithms to compose new musical structures. There's even a great, humorous article on how to use offline technology for enhancing your online sentience – and more!
And here's a quote from the introduction to my chapter.
Some composers have explored the music that arises from mathematical functions, such as fractals. Composers such as myself have tried to use computers not just to imitate the human creative process, but also to simulate the possibility of inhuman creativity. This has involved employing models of intelligence and computation that aren't based on cognition, such as cellular automata, genetic algorithms and the topic of this article, swarm intelligence. The most difficult problem with using any of these systems in music is that they aren't inherently musical. In general, they are inherently unrelated to music. To write music using data from an arbitrary process, the composer must find a way of translating the non-musical data into musical data. The problem of mapping a process from one domain to work in an entirely unrelated domain is called the mapping problem. In this article, the problem is mapping from a virtual swarm to music, however, the problem applies in similar ways to algorithmic art in general. Some algorithms may be easily translated into one type of art or music, while other algorithms may require complex math for even basic art to emerge.
Sound Synthesis in Java
Sound Synthesis in Java introduces sound synthesis concepts using the most widely taught programming language in the world, Java. Using the Beads library, it walks readers through the basics of sound generating programs all the way up through imitations of commercial synthesizers. In eleven chapters the book covers additive synthesis, modulation synthesis, subtractive synthesis, granular synthesis, MIDI keyboard input, rendering to audio files and more. Each chapter includes an explanation of the topic and examples that are as simple as possible so even beginning programmers can follow along. Part two of the book includes six projects that show the reader how to build arpeggiators, imitate an analog synthesizer, and create flowing soundscapes using granular synthesis.
Sonifying Processing is available for free online.
Read Online
Read Sound Synthesis in Java online. The source code is available from links in the text.
Make Explosive Soundscapes with Circular Sound
I've just finished work on a new musical instrument for Android devices. It's called Circular Sound, and it's aimed at people who like noise.
Circular Sound is similar to my other recent mobile instruments in that it combines a sampler with custom digital signal processing, and a unique interface. Sounds can be loaded from a configurable directory on the device, or you can play around with the default sounds, which are from freesound.org. Then they are loaded into spheres that are arranged in a circle on the left half of the screen. The left half of the screen is the source audio mixer, while the right half is used to control effects. The effects include waveshaping, granulation, delay, and modulation.
The goal of Circular Sound is to give a simple access point into generating various types of noise that is related in some way to the source sounds provided by the user.
Download it for free on Google Play and shoot me a comment to let me know if you make something cool with it!
UPDATE: I have been unable to continue maintaining Circular Sound, so it is no longer available.
Why are Side Effects Bad?
Side effects are any observable change to the state of an object. Some people might qualify this by saying that side effects are implicit or unintended changes to state.
Mutator methods, or setters, which are designed to change the state of an object, should have side effects. Accessor methods, or getters, should not have side effects. Other methods should generally try to avoid changing state beyond what is necessary for the intended task.
Why are these guidelines best practices though? What makes side effects so bad? Students have asked me this question many times, and I've worked with many experienced programmers who don't seem to understand why minimizing side effects is a good goal.
For example, I recently commented out a block of my peer's code because it had detrimental side effects. The code was intended to add color highlighting to log entries to make his debugging easier. The problem with the code was that the syntax highlighting xml was leaking into our save files. He was applying the highlight to a key, then using that key both in the log and in the save file. Worse still, he wrote this code so that it only occurred on some platforms. When I was debugging a cross platform feature, I got very unpredictable behavior and ultimately traced it back to this block.
This is an example where code was intended for one purpose, but it also did something else. That something else is the side effect, and you can see how it caused problems for other developers. Since his change was undocumented and uncommented, I spent hours tracking it down. As a team, we lost significant productivity due to a side effect.
Side effects are bad because they make a code base less agile. Side effects cause bugs that are difficult to find, and lead to code that is more difficult to maintain.
Before I continue, let me be clear that all code style guidelines should be broken sometimes. For each situation, a guideline or design pattern may be better or worse, and I recognize that we are always working in shades of gray.
Generally, a block of code should be written for one purpose. If it is a method, then it should do one thing. If another thing needs to be done with an object, then that should be encapsulated in another method.
Here's a hypothetical example that I've seen played out hundreds of times in my career.
A class needs to do task X. A programmer may write a method to do task X, but he accidentally includes logic that also does task Y. Later, he may see that he needs to do task Y all alone. So he writes a method to do task Y. That's where the problem is compounded.
Later still, the definition of task Y changes. So another programmer has to rewrite task Y. He goes to the class, changes the method for task Y alone, does a few quick tests, and proceeds on his merry way.
Then mysterious bugs start occurring. QA can't really track them down to one thing because they only occur sporadically after task Y. Finally, it takes many man-hours to remove task Y from the method for task X.
In this example, the side effect led to code duplication, which led to trouble when updating the code, which led to bugs that cost many hours to track down. The fewer side effects you introduce, the easier your code will be to maintain.
These two examples show how side effects can derail development and why they are so inimical. We all write code with side effect occasionally, but it's our job to figure out how to do it in a way that doesn't make the code more difficult to maintain.
How Can Comments be Controversial?
Comments add information to your code. They don't impact the execution of that code. So how can they be bad? I believe that more commenting is better, and that comments are a vital tool in maintaining an agile codebase, but I've met many experienced developers who argue with me about comments.
The Pragmatic Programmer is an excellent book. It outlines best practices that have solidified over the last thirty years or so as software development has grown as a field. The authors share these practices as a series of tips that are justified by real world anecdotes. In a section on comments the authors say
"The DRY principle tells us to keep the low-level knowledge in the code, where it belongs, and reserve the comments for other, high-level explanations. Otherwise, we're duplicating knowledge, and every change means changing both the code and the comments. The comments will inevitably become out of date, and untrustworthy comments are worse than no comments at all"
They make the point that obsolete comments are worse than no comments at all. This is undoubtably true, but it has been bastardized by many programmers as an excuse for not commenting. I've heard this excuse, along with two others to justify a lack of comments in code.
- The code is self-commenting.
- Out of date comments are worse than no comments.
- You shouldn't rely on comments.
In this post I want to address each of these points. I think that thorough commenting speeds up the rate of consumption of code, and that obsolete comments are a failure in maintenance, rather than in comments in general.
1. The code is self-commenting
This is the excuse I hear most often from developers who don't comment their code. The thinking is that clear variable and method names replace the need for comments. There are situations where this is true. For very short methods with a single purpose, I see that no additional comments may be necessary.
But most methods are not very short, and most methods achieve several things that relate to one overall purpose. So in the first place, very few methods fit the bill.
This is not a valid complaint because good comments only make the code easier to read. While the code may be legible on its own, a comment that explains the purpose of a method, or the reason for a string operation, can only make it easier to read. The author of Clean Code agrees due to the amount of time we spend reading code.
"the ratio of time spent reading vs. writing is well over 10:1. We are constantly reading old code as part of the effort to write new code. Because this ratio is so high, we want the reading of code to be easy, even if it makes the writing harder."
Because we spend more time reading code than writing it, a responsible programmer knows that the extra comment in a method which takes very little time to write may well save another programmer many hours of learning.
2. Out of date comments are worse than no comments
To refute this argument, lets take cars as an analogy. If you buy a car, then drive it for twenty thousand miles without an oil change, it will break down. If you did that, the fault wouldn't be in cars in general. Rather, the fault is in the lack of maintenance.
Similary, obsolete comments are not a failure of comments in general, but a failure of the programmer who updated the code without updating the comments. That other programmers have bad practices does not justify continuing that bad practice.
Obsolete comments are a broken window. If you see them, it is your job to fix them.
So while obsolete comments ARE worse than no comments, that argument is an argument against bad coding practices, not comments in general.
3. You shouldn't rely on comments
The argument that a programmer shouldn't rely on comments is a subtle dig at another programmer's skill. They're saying "a good programmer can understand this code with no comments."
There is a grain of a valid point there. If you consider yourself a seasoned programmer, then you should be accustomed to wading through large volumes of old code and parsing out the real meaning and functionality.
But even for veteran programmers, comments make it easier to read and understand the code. It's much easier to interpret a one line summary of a block of code than to parse that block line by line. It's much easier to store that one line in your head than it is to store the whole block in your head.
The real reason why this is the worst of excuses is that not all programmers working on your code are senior programmers. Some are green graduates straight out of college. Yes, a more experienced programmer could understand the code without comments, but it will cost that graduate hours to figure out what they could have gotten in minutes with good comments.
My Commenting Principle
Comment your code so that a green programmer can understand it.
Or…
Comment your code so that your boss can understand it.
What does it take to make your code readable by a green graduate? Imagine them going through it line by line. If there are no comments, then they need to understand every function call, and every library routine. They need to understand a bit about your architecture and date model.
Now imagine that every single line is commented.
In that scenario, a green programmer can read your file by reading one comment after the next. If they need more information, they can still drill into the code and get the details, but they can understand the overview just through comments.
The closer you can get to that, the better.
Don't listen to the arguments against thorough commenting habits. Just point the detractors here.
Why I Created QuizMana.com
When I was teaching at UCSC and SJSU, I taught very large courses, often consisting of over 200 students. Courses that size create lots of infrastructural problems. Communication is a huge problem, but that's mostly taken care of by Canvas or Blackboard. The bigger problem for me was assessments.
I think that even well-written multiple choice quizzes are not great assessments. In multiple choice quizzes, students always have the answer presented to them. Even very difficult multiple choice quizzes never ask students to explain their reasoning. They never force students to come up with a concept from memory. They present students with the false premise that knowledge is fixed, and is simply a matter of choosing from what already exists.
So I always wanted to use other types of questions, and I did. Or at least I tried to. To incorporate even one short answer question meant hours of grading. I once did a single essay assignment for a large class, and probably spent over forty hours grading essays.
I needed a grading tool that could quickly save me time, while allowing me to diversify the types of assessments I could use on a week-to-week basis. For the past three months I've been building a website that automatically grades short answer, essay, and multiple choice questions. I call it QuizMana.com, and it's accepting applicants for the closed beta right now.
The hook of the website is automatic grading, but the purpose of it is to save teachers as much time as possible. So the grader gives a score for every question, but the site then points the teacher to all the "low confidence" scores. These are the scores where the grader didn't have enough information to be fully confident it gave an accurate score. The teacher can then grade the low confidence responses on a single page, and then they are done!
So the grader is part of a larger process that means that teachers only grade the work that actually needs their attention.
I think this can be a great tool, and I think it will save teachers a lot of time. If you're teaching a large class this semester, then give yourself a new tool, and try QuizMana.
The Simplest Unity Bundling Example I Could Make
Asset bundles in Unity can be tricky. I've spent a lot of time at work developing a pretty slick bundling system. The system pulls the latest from our git repo, then uses manifests to pull in new procedural assets, then generates all the bundles for each level, then uploads them into versioned folders on Amazon S3.
Most big projects will need something like this.
But it's probably best to start with something simpler. The system I've built for work is probably 500 – 1000 lines of code with everything included. This weekend I wanted to find the least amount of code with which I could make a working bundles system.
This system has no extra features. It doesn't support simulation mode. It doesn't support procedural assets. It doesn't support multiple simultaneous downloads. It doesn't support viewing download progress. But it does work.
Here's how to get Unity asset bundles working in under 200 lines of code.
Download the AssetBundleManager from the Asset Store
You don't need the examples. You're really only using this for the UI features added to the Editor, but some of the included classes in that package are also a good starting point.
Assign assets to bundles
When you select an asset, you should see a drop down all the way at the very bottom of the inspector. Use that dropdown to create a new asset bundle, then add your assets to it.
Create editor script for clearing bundle cache
When you are testing bundles, you will need an extra menu item to delete your local cache. Make an Editor folder somewhere and drop this script in it.
using UnityEngine;
using UnityEditor;
public class BundleOptions
{
///
/// Delete all cached bundles. This is necessary for testing.
///
[MenuItem("Assets/AssetBundles/Clear Bundle Cache")]
static void LogSelectedTransformName()
{
if( Caching.CleanCache() )
{
Debug.Log("All asset bundles deleted from cache!");
}
else
{
Debug.LogWarning("Unable to delete cached bundles! Are any bundles in use?");
}
}
}
You will want to click Assets > AssetBundles > Clear Bundle Cache between each generation of bundles to make sure that you are using the latest version in editor.
Create a BundleManager class
Your BundleManager class will download bundles and allow the program to access asset bundles. If you expand this example, then this will be the center of your work.
using UnityEngine;
using System.Collections;
using System.Collections.Generic;
public class BundleManager : MonoBehaviour
{
// the web server where your bundles are stored - you can test this locally using file://c:/path-to-bundles/etc
#if UNITY_EDITOR
private static string baseBundleUrl = "file://c:/example/Unity/MyGame/AssetBundles";
#else
private static string baseBundleUrl = "http://www.example.com/MyGame/AssetBundles";
#endif
// this dictionary will store references to each downloaded asset bundle
private static Dictionary bundles;
///
/// Get a reference to an AssetBundle instance that has already been downloaded.
///
/// This should be used when you want to load an asset from a bundle.
///
/// the name of the bundle to get
/// a reference to an AssetBundle or null if that bundle has not yet been downloaded or cached
public static AssetBundle GetBundle(string name)
{
if( bundles != null && bundles.ContainsKey(name) )
{
return bundles[name];
}
return null;
}
///
/// Very simple method for downloading a single bundle.
/// Extensions to this method may include a progress monitor and a callback for when the download is complete.
///
/// the name of the bundle you want to download
/// an IEnumerator to be run as a Coroutine
public static IEnumerator DownloadBundle(string bundleName, System.Action downloadCallback = null)
{
Debug.Log("Attempting to load bundle " + bundleName + "...");
WWW www = WWW.LoadFromCacheOrDownload(getFullBundleUrl(bundleName), 1);
yield return www;
// if there was a download error, then log it
if( !string.IsNullOrEmpty(www.error) )
{
Debug.LogError("Error while downloading bundle " + bundleName + ": " + www.error);
}
else // if there was no download error
{
// try to get the downloaded bundle
AssetBundle newBundle = www.assetBundle;
// if the bundle is null, then log an error
if( newBundle == null )
{
Debug.LogError("Unable to save bundle " + bundleName + ": Bundle is null!");
}
else // if a valid bundle was downloaded
{
if (bundles == null)
bundles = new Dictionary();
// store a reference to that bundle in the dictionary
bundles[bundleName] = newBundle;
Debug.Log("Successfully loaded " + bundleName + ".");
}
}
// if there is a downloadCallback, then call it
if( downloadCallback != null )
{
downloadCallback();
}
}
///
/// Return a string representation of the platform that will go into the bundle path.
///
private static string getBundlePlatform()
{
if (Application.platform == RuntimePlatform.Android)
{
return "Android";
}
else if( Application.platform == RuntimePlatform.WindowsEditor )
{
return "Windows";
}
// maybe this is some strange version of Android? Need this for Kindle.
return "Android";
// do not support other platforms
//throw new System.Exception("This platform is not supported!");
}
private static string getFullBundleUrl(string bundleName)
{
return baseBundleUrl + "/" + getBundlePlatform() + "/" + bundleName;
}
// Use this to initialize the bundles dictionary
void Start ()
{
if (bundles == null)
bundles = new Dictionary();
}
}
Modify that script by replacing baseBundleUrl with the place where your bundles are stored locally for development and remotely for production.
You will also need to hook this script into the program somehow. You can attach it to a GameObject if you like, but that isn't strictly necessary. All you really need to do is start a coroutine to run the DownloadBundle method, then call GetBundle to get the download. Then just call the LoadAsset method on the asset you want to instantiate.
Generate bundles
When you click Assets > AssetBundles > Build AssetBundles, it will generate bundles for whatever platform is selected. You will need to generate bundles for your development machine and for your target platform. Make sure to modify the getBundlePlatform method to support your platform.
And…. ?
And that's it. Asset bundles are not really that complex when you boil them down. For my music apps, I just want them to store a few extra sound files that I don't want to distribute with the executable. So a system like this works fine. I just start the download when the app starts.
Of course, for larger projects you will need many more features. Versioning bundles is very important for the development and build processes. Also you will want to show a progress bar on the screen. And you will want to load entire levels into bundles, which is a little more tricky. But hopefully this will get you started on the right track.
How to Share an Audio File on Android from Unity/C#
Rendering audio to a file is an important feature of an audio synthesizer, but if the user can't share the file, then it's not very useful. In my second pass on my synthesizers, I'm adding the ability to share rendered audio files using email or text message.
The code for sharing audio files is tricky. You have to tell Unity to generate some Java code that launches something called an Intent. So this code basically instantiates the Java classes for the Intent and the File, then starts the activity for the intent.
Figuring out the code is tough, but you also need to change a setting in your player settings. Specifically, I couldn't get this code to work without Write Access: External (SDCard) enabled in Player Settings. Even if I am writing to internal storage only, I need to tell Unity to request external write access. I'm assuming that the extra privileges are needed for sharing the file.
Here's the code.
public static void ShareAndroid(string path)
{
// create the Android/Java Intent objects
AndroidJavaClass intentClass = new AndroidJavaClass("android.content.Intent");
AndroidJavaObject intentObject = new AndroidJavaObject("android.content.Intent");
// set properties of the intent
intentObject.Call("setAction", intentClass.GetStatic("ACTION_SEND"));
intentObject.Call("setType", "*/*");
//instantiate the class Uri
AndroidJavaClass uriClass = new AndroidJavaClass("android.net.Uri");
// log the attach path
Debug.Log("Attempting to attach file://" + path);
// check if the file exists
AndroidJavaClass fileClass = new AndroidJavaClass("java.io.File");
AndroidJavaObject fileObject = new AndroidJavaObject("java.io.File", path);// Set Image Path Here
//instantiate the object Uri with the parse of the url's file
AndroidJavaObject uriObject = uriClass.CallStatic("parse", "file://" + path);
// call the exists method on the File object
bool fileExist = fileObject.Call("exists");
Debug.Log("File exists: " + fileExist);
// attach the Uri instance to the intent
intentObject.Call("putExtra", intentClass.GetStatic("EXTRA_STREAM"), uriObject);
// instantiate the current activity
AndroidJavaClass unity = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
AndroidJavaObject currentActivity = unity.GetStatic("currentActivity");
// start the new intent - for this to work, you must have Write Access: External (SDCard) enabled in Player Settings!
currentActivity.Call("startActivity", intentObject);
}
Recording In-Game Audio in Unity
Recently I began doing a second pass on my synthesizers in the Google Play store. I think the core of each of those synths is pretty solid, but they are still missing some key features. For example, if you want to record a performance, you must record the output of the headphone jack.
So I just finished writing a class that renders a Unity audio stream to a wave file, and I wanted to share it here.
The class is called AudioRenderer. It's a MonoBehaviour that uses the OnAudioFilterRead method to write chunks of data to a stream. When the performance ends, the Save method is used to save to a canonical wav file.
The full AudioRenderer class is pasted here. As written it will only work on 16bit/44kHz audio streams, but it should be easily adaptable.
using UnityEngine;
using System;
using System.IO;
public class AudioRenderer : MonoBehaviour
{
#region Fields, Properties, and Inner Classes
// constants for the wave file header
private const int HEADER_SIZE = 44;
private const short BITS_PER_SAMPLE = 16;
private const int SAMPLE_RATE = 44100;
// the number of audio channels in the output file
private int channels = 2;
// the audio stream instance
private MemoryStream outputStream;
private BinaryWriter outputWriter;
// should this object be rendering to the output stream?
public bool Rendering = false;
/// The status of a render
public enum Status
{
UNKNOWN,
SUCCESS,
FAIL,
ASYNC
}
/// The result of a render.
public class Result
{
public Status State;
public string Message;
public Result(Status newState = Status.UNKNOWN, string newMessage = "")
{
this.State = newState;
this.Message = newMessage;
}
}
#endregion
public AudioRenderer()
{
this.Clear();
}
// reset the renderer
public void Clear()
{
this.outputStream = new MemoryStream();
this.outputWriter = new BinaryWriter(outputStream);
}
/// Write a chunk of data to the output stream.
public void Write(float[] audioData)
{
// Convert numeric audio data to bytes
for (int i = 0; i < audioData.Length; i++)
{
// write the short to the stream
this.outputWriter.Write((short)(audioData[i] * (float)Int16.MaxValue));
}
}
// write the incoming audio to the output string
void OnAudioFilterRead(float[] data, int channels)
{
if( this.Rendering )
{
// store the number of channels we are rendering
this.channels = channels;
// store the data stream
this.Write(data);
}
}
#region File I/O
public AudioRenderer.Result Save(string filename)
{
Result result = new AudioRenderer.Result();
if (outputStream.Length > 0)
{
// add a header to the file so we can send it to the SoundPlayer
this.AddHeader();
// if a filename was passed in
if (filename.Length > 0)
{
// Save to a file. Print a warning if overwriting a file.
if (File.Exists(filename))
Debug.LogWarning("Overwriting " + filename + "...");
// reset the stream pointer to the beginning of the stream
outputStream.Position = 0;
// write the stream to a file
FileStream fs = File.OpenWrite(filename);
this.outputStream.WriteTo(fs);
fs.Close();
// for debugging only
Debug.Log("Finished saving to " + filename + ".");
}
result.State = Status.SUCCESS;
}
else
{
Debug.LogWarning("There is no audio data to save!");
result.State = Status.FAIL;
result.Message = "There is no audio data to save!";
}
return result;
}
/// This generates a simple header for a canonical wave file,
/// which is the simplest practical audio file format. It
/// writes the header and the audio file to a new stream, then
/// moves the reference to that stream.
///
/// See this page for details on canonical wave files:
/// http://www.lightlink.com/tjweber/StripWav/Canon.html
private void AddHeader()
{
// reset the output stream
outputStream.Position = 0;
// calculate the number of samples in the data chunk
long numberOfSamples = outputStream.Length / (BITS_PER_SAMPLE / 8);
// create a new MemoryStream that will have both the audio data AND the header
MemoryStream newOutputStream = new MemoryStream();
BinaryWriter writer = new BinaryWriter(newOutputStream);
writer.Write(0x46464952); // "RIFF" in ASCII
// write the number of bytes in the entire file
writer.Write((int)(HEADER_SIZE + (numberOfSamples * BITS_PER_SAMPLE * channels / 8)) - 8);
writer.Write(0x45564157); // "WAVE" in ASCII
writer.Write(0x20746d66); // "fmt " in ASCII
writer.Write(16);
// write the format tag. 1 = PCM
writer.Write((short)1);
// write the number of channels.
writer.Write((short)channels);
// write the sample rate. 44100 in this case. The number of audio samples per second
writer.Write(SAMPLE_RATE);
writer.Write(SAMPLE_RATE * channels * (BITS_PER_SAMPLE / 8));
writer.Write((short)(channels * (BITS_PER_SAMPLE / 8)));
// 16 bits per sample
writer.Write(BITS_PER_SAMPLE);
// "data" in ASCII. Start the data chunk.
writer.Write(0x61746164);
// write the number of bytes in the data portion
writer.Write((int)(numberOfSamples * BITS_PER_SAMPLE * channels / 8));
// copy over the actual audio data
this.outputStream.WriteTo(newOutputStream);
// move the reference to the new stream
this.outputStream = newOutputStream;
}
#endregion
}
When Code Duplication is not Code Duplication
Duplicating code is a bad thing. Any engineer worth his salt knows that the more you repeat yourself, the more difficult it will be to maintain your code. We've enshrined this in a well-known principle called the DRY principle, where DRY is an acronym standing for Don't Repeat Yourself. So code duplication should be avoided at all costs. Right?
At work I recently came across an interesting case of code duplication that merits more thought, and shows how there is some subtlety needed in application of every coding guideline, even the bedrock ones.
Consider the following CSS, which is a simplified version of a common scenario.
.title {
color: #111111;
}
.text-color-gray-1 {
color: #111111;
}
This looks like code duplication, right? If both classes are applying the same color, then they do the same thing. If the do the same thing, then they should BE the same thing, right?
But CSS and markup in general presents an interesting case. Are these rules really doing the same thing? Are they both responsible for making the text gray? No.
The function of these two rules is different, even though the effect is the same. The first rule styles titles on the website, while the second rule styles any arbitrary div. The first rule is a generalized style, while the second rule is a special case override. The two rules do fundamentally different things.
Imagine a case where we optimized those two classes by removing the title class and just using the latter class. Then the designer changes the title color to dark blue. To change the title color, the developer now has to replace each occurrence of .text-color-gray-1 where it styles a title. So, by optimizing two things with different purposes, the developer has actually made more work.
It's important to recognize in this case that code duplication is not always code duplication. Just because these two CSS classes are applying the same color doesn't mean that they are doing the same thing. In this case, the CSS classes are more like variables than methods. They hold the same value, but that is just a coincidence.
What looks like code duplication is not actually code duplication.
But… what is the correct thing?
There is no right answer here. It's a complex problem. You could solve it in lots of different ways, and there are probably three or four different approaches that are equally valid, in the sense that they result in the same amount of maintenance.
The important thing is not to insist that there is one right way to solve this problem, but to recognize that blithely applying the DRY principle here may not be the path to less maintenance.
Pride in Software Craftsmanship
As I spend more and more time in Silicon Valley, my views on software management are changing. I read Radical Candor recently, and while I agree with everything in it, I feel like it over-complicates things.
This meditation has been pushed in part by my passion for food. I like going to new restaurants. It brings me joy to try something new, even if it's not a restaurant that would ever be considered for a Michelin Star. Even crappy looking restaurants can serve great food.
I am often awed by the disconnect between various parts of the restaurant business and the quality of the food. Some restaurants are spotlessly clean, have have beautiful decor, and amazing service… but the food is mediocre. The menu is bland and uninspired, and the food itself is prepared with all the zeal that a minimum wage employee can manage.
Then I'll go to a dirty looking greek joint down the road, and the service will be awful… but the menu is inspired. It's not the standard "greek" menu, but it's got little variations on the dishes. And when the food comes out (finally), maybe it isn't beautiful on the plate, but the flavors come together to make something greater than the ingredients and the recipe.
What seems to distinguish a good restaurant from a crappy one is pride. At restaurants that I return to, there is someone there, maybe a manager, maybe a cook, maybe the chef who designed the menu, who takes great pride in his work.
There's a diner by my old house, for instance, where the food is … diner food. There's no reason to go back to the restaurant… except for the manager. The man who runs the floor, seats the patrons, deals with the kitchen, and does all the little things that make a restaurant tick. He manages to make that particular diner worth going to. And for a guy who has two young kids, that's terrific.
I am starting to think that the same basic principle applies to software engineers. I've met brilliant engineers with all sorts of characteristics. Some of them have a lot of education and read all the latest guides. Others have little education, and don't read at all. The main thing that makes them good engineers is that they take pride in their work. They care about the quality of their work, regardless of how many people are going to use it, or how much time they put into it. They write quality code because their work matters.
So when it comes to managing software projects, I'm starting to think that all of these systems boil down to two basic steps.
- Put your engineers in a position to take pride in their work.
- Get out of the way.
Obviously, the first step is non-trivial. It's why there are so many books on the topic. But at the end of the day, pride is what matters.
Tips for Managing Joins in Looker
Looker is a fantastic product. It really makes data and visualizations much more manageable. The main goal of Looker is to allow people who aren't data analysts to do some basic data analysis. To some extent, it achieves this, but there are limits to how far this can go. Ultimately, Looker is a big graphical user interface for writing SQL and generating charts. Under-the-hood, it's programmable by data engineers, but it's limited by the fact that non-technical users are using it.
The major design challenge for Looker is joins. A data engineer writes the joins into what Looker calls "explores". Explores are rules for how data can be explored, but ultimately just a container for joins. When someone creates a new chart, they start by selecting an explore, and thus selecting the joins that will be used in the chart.
They pick the join from a dropdown under the word "Explore". This is the main design bottleneck. Such a UI encourages users to have only a limited number of joins that can fit in the vertical resolution of the screen. This means limiting the number of explores, and hence limiting the ways tables are joined. This encourages using pre-existing joins for new charts.
This creates two problems.
- A non-technical user will not understand the implication of choosing an explore. They may not see that the explore they chose limits how the data can be analyzed. In fact, a non-savvy user may pick the wrong explore entirely, and create a chart that is entirely wrong.
- The joins may evolve over time. A programmer might change a join for a new chart, and this may make old charts incorrect.
The problem is that SQL joins are fundamentally interpretations of the data. Unless a join occurs on id fields AND is a one-to-one relationship, then a join interprets the data in some way.
So how can you limit the negative impact of re-using joins?
1. Encourage simple charts
Encourage your teammates to make charts as simple as possible. If possible, a chart should show a single quantity as it changes over a single dimension. This should eliminate or minimize the use of joins in the chart, thus making it far more future-proof.
2. Give explores long, verbose names
Make explore names as descriptive as possible. Try to communicate the choice that a user is making when they choose an explore. For instance, you might name one explore "Products Today" and another one "Product Events Over Time". These names might indicate that the first explore looks at the products table, but the second explore shows events relating to products joined with a time dimension.
One of the mistakes I made while first starting out with Looker is naming the explores with single word names. I now see that short names create maintenance nightmares. Before assessing the problems with a given chart, I need to know which explore the maker chose for it, and because the names were selected so poorly, the choice was often incorrect.
I hope these ideas help you find a path to a maintainable data project. To be honest, I have a lot of digging-out to do!
Glitching Images in Processing
This summer I'm going to release a new album of solo electronic music that is heavily influenced by EDM and classic rock. For the past few weeks I've been trying to figure out what to do about the art on the new album.
The album is called "FYNIX Fights Back" so I decided to use images of animals fighting, but I didn't just want to pull up generic images. I wanted to add my own special touch to each image. So I pulled out a tool that I haven't used in years: the Processing programming language.
Processing is great for simple algorithmic art. In this case I wanted to glitch some images interactively, but I wasn't exactly sure what I wanted to do.
So I just started experimenting. I played with colors and shapes and randomness. I like to derive randomess based on mouse movement. The more the mouse moves, the more randomness is imparted to whatever the user is doing.
I added image glitching based on mouse speed. The quicker the cursor moves, the more random are the generated shapes, and the more they are offset from their source position in the original image.
Here's the end result.
Here's the source code. Make it even better.
import java.awt.event.KeyEvent;
// relationship between image size and canvas size
float scalingFactor = 2.0;
// image from unsplash.com
String imageFilename = "YOUR_FILE_IN_DATA_FOLDER.jpg";
// image container
PImage img;
PImage scaledImage;
int minimumVelocity = 40;
int velocityFactorDivisor = 5; // the larger this is, the more vertices you will get
int velocityFactor = 1; // this will be overridden below
float minimumImageSize = 10.0f;
boolean firstDraw = true;
int currentImage = 1;
void settings() {
// load the source image
img = loadImage(imageFilename);
// load the pixel colors into the pixels array
img.loadPixels();
// create a canvas that is proportional to the selected image
size((int)(img.width / scalingFactor), (int)(img.height / scalingFactor));
// scale the image for the window size
scaledImage = loadImage(imageFilename);
scaledImage.resize(width, height);
// override velocityFactor
velocityFactor = (int)(width / velocityFactorDivisor);
}
void setup() {
// disable lines
noStroke();
}
void keyPressed() {
if(keyCode == KeyEvent.VK_R) {
firstDraw = true; // if the user presses ENTER, then reset
}
}
void draw() {
if(firstDraw) {
image(scaledImage, 0, 0);
firstDraw = false;
}
// right click to render to image file
if(mousePressed && mouseButton == RIGHT) {
save(imageFilename.replace(".jpg", "") + "_render_" + currentImage + ".tga");
currentImage++;
}
if(mousePressed && mouseButton == LEFT && mouseX >= 0 && mouseX < width && mouseY >= 0 && mouseY < height) {
int velocityX = minimumVelocity + (3 * velocity(mouseX, pmouseX, width));
int velocityY = minimumVelocity + (3 * velocity(mouseY, pmouseY, height));
color c = img.pixels[mousePositionToPixelCoordinate(mouseX, mouseY)];
int vertexCount = ((3 * velocityFactor) + velocityX + velocityY) / velocityFactor;
int minimumX = mouseX - (velocityX / 2);
int maximumX = mouseX + (velocityX / 2);
int minimumY = mouseY - (velocityY / 2);
int maximumY = mouseY + (velocityY / 2);
PGraphics pg = createGraphics(maximumX - minimumX, maximumY - minimumY);
pg.beginDraw();
pg.noStroke();
pg.fill(c);
// first draw a shape into the buffer
pg.beginShape();
for(int i = 0; i < vertexCount; i++) {
pg.vertex(random(0, pg.width), random(0, pg.height));
}
pg.endShape();
pg.endDraw();
pg.loadPixels();
// then copy image pixels into the shape
// get the upper left coordinate in the source image
int startingCoordinateInSourceImage = mousePositionToPixelCoordinate(minimumX, minimumY);
// get the width of the source image
int sourceImageWidth = (int)(img.width);
// set the offset from the source image
int offsetX = velocity(mouseX, pmouseX, width);
int offsetY = velocity(mouseY, pmouseY, height);
// ensure that the offset doesn't go off the canvas
if(mouseX > width / 2) offsetX *= -1;
if(mouseY > height / 2) offsetY *= -1;
for(int y = 0; y < pg.height; y++) {
for(int x = 0; x < pg.width; x++) {
// calculate the coordinate in the destination image
int newImageY = y * pg.width;
int newImageX = x;
int newImageCoordinate = newImageX + newImageY;
// calculate the location in the source image
//int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + (x * scalingFactor) + ((y * scalingFactor) * sourceImageWidth));
//int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + ((x + offsetX) * scalingFactor) + (((y + offsetY) * scalingFactor) * sourceImageWidth));
int sourceImageX = (int)(((x + offsetX) * scalingFactor));
int sourceImageY = (int)((((y + offsetY) * scalingFactor) * sourceImageWidth));
int sourceImageCoordinate = (int)(startingCoordinateInSourceImage + sourceImageX + sourceImageY);
// ensure the calculated coordinates are within bounds
if(newImageCoordinate > 0 && newImageCoordinate < pg.pixels.length
&& sourceImageCoordinate > 0 && sourceImageCoordinate < img.pixels.length
&& pg.pixels[newImageCoordinate] == c) {
pg.pixels[newImageCoordinate] = img.pixels[sourceImageCoordinate];
}
}
}
image(pg, minimumX, minimumY);
}
}
// convert the mouse position to a coordinate within the source image
int mousePositionToPixelCoordinate(int mouseX, int mouseY) {
return (int)(mouseX * scalingFactor) + (int)((mouseY * scalingFactor) * img.width);
}
// This sort of calculates mouse velocity as relative to canvas size
int velocity(float x1, float x2, float size) {
int val = (int)(Math.abs((x1 - x2) / size) * size);
return val;
}
How to render synchronous audio and video in Processing in 2021
About a decade ago I wrote a blog post about rendering synchronous audio and video in processing. I posted it on my now-defunct blog, computermusicblog.com. Recently, I searched for the same topic, and found that my old post was one of the top hits, but my old blog was gone.
So in this post I want to give searchers an updated guide for rendering synchronous audio and video in processing.
It's still a headache to render synchronous audio and video in Processing, but with the technique here you should be able to copy my work and create a simple 2-click process that will get you the results you want in under 100 lines of code.
Prerequisites
You must install Processing, Minim, VideoExport, and ffmpeg on your computer. Processing can be installed from processing.org/. Minim and VideoExport are Processing libraries that you can add via Processing menus (Sketch > Import Library > Add Library). You must add ffmpeg to your path. Google how to do that.
The final, crappy prerequisite for this particular tutorial is that you must be working with a pre-rendered wav file. In other words, this will work for generating Processing visuals that are based on an audio file, but not for Processing sketches that synthesize video and audio at the same time.
Overview
Here's what the overall process looks like.
- Run the Processing sketch. Press q to quit and render the video file.
- Run ffmpeg to combine the source audio file with the rendered video.
Source Code
Without further ado, here's the source. This code is a simple audio visualizer that paints the waveform over a background image. Notice the ffmpeg instructions in the long comment at the top.
/*
This is a basic audio visualizer created using Processing.
Press q to quit and render the video.
For more information about Minim, see http://code.compartmental.net/tools/minim/quickstart/
For more information about VideoExport, see https://timrodenbroeker.de/processing-tutorial-video-export/
Use ffmpeg to combine the source audio with the rendered video.
See https://superuser.com/questions/277642/how-to-merge-audio-and-video-file-in-ffmpeg
The command will look something like this:
ffmpeg -i render.mp4 -i data/audio.wav -c:v copy -c:a aac -shortest output.mp4
I prefer to add ffmpeg to my path (google how to do this), then put the above command
into a batch file.
*/
// Minim for playing audio files
import ddf.minim.*;
// VideoExport for rendering videos
import com.hamoid.*;
// Audio related objects
Minim minim;
AudioPlayer song;
String audioFile = "audio.wav"; // The filename for your music. Must be a 16 bit wav file. Use Audacity to convert.
// image related objects
float scaleFactor = 0.25f; // Multiplied by the image size to set the canvas size. Changing this is how you change the resolution of the sketch.
int middleY = 0; // this will be overridden in setup
PImage background; // the background image
String imageFile = "background.jpg"; // The filename for your background image. The file must be present in the data folder for your sketch.
// video related objects
int frameRate = 24; // This framerate MUST be achievable by your computer. Consider lowering the resolution.
VideoExport videoExport;
public void settings() {
background = loadImage(imageFile);
// set the size of the canvas window based on the loaded image
size((int)(background.width * scaleFactor), (int)(background.height * scaleFactor));
}
void setup() {
frameRate(frameRate);
videoExport = new VideoExport(this, "render.mp4");
videoExport.setFrameRate(frameRate);
videoExport.startMovie();
minim = new Minim(this);
// the second param sets the buffer size to the width of the canvas
song = minim.loadFile(audioFile, width);
middleY = height / 2;
if(song != null) {
song.play();
}
fill(255);
stroke(255);
strokeWeight(2);
// tell Processing to draw images semi-transparent
tint(255, 255, 255, 80);
}
void draw() {
image(background, 0, 0, width, height);
for(int i = 0; i < song.bufferSize() - 1; i++) {
line(i, middleY + (song.mix.get(i) * middleY), i+1, middleY + (song.mix.get(i+1) * middleY));
}
videoExport.saveFrame(); // render a video frame
}
void keyPressed() {
if (key == 'q') {
videoExport.endMovie(); // Render a silent mp4 video.
exit();
}
}
Composing with All Sound
I sometimes get asked about my dissertation, so I wanted to write a blog post to explain it. In this post, I describe the work I did for my dissertation, which brought together web APIs and a network model of creativity.
Composing with all sound
When I was in graduate school I was somewhat obsessed with the idea of composing with all sound. When I say "composing with all sound" I don't mean composing with a lot of sounds. I mean literally all sound that exists. Composing with all sounds that have ever existed is literally impossible, but the internet gives us a treasure trove of sound.
So I looked at the largest collections of sound on the internet, and the best that I found in 2011 was the website freesound.org/. Freesound is great for several reasons.
- The library is absolutely massive and constantly growing
- They have a free to use API
- Users can upload tags and descriptions of the sounds
- Freesound analyzes the sounds and gives access to those descriptors via the API
A model of creativity
Once I had a source for sounds, all I needed was a way to connect them. Neural network research was blossoming in 2011, so I tried to find a neural model that I could connect to the data on Freesound. That's when I found Melissa Schilling's network model of cognitive insight.
Schilling's theory essentially sees ideas as networks in the brain. Ideas are connected by various relationships. Ideas can look alike, sound alike, share a similar space, be described by similar words, and so on. In Schilling's model, cognitive insight, or creativity, occurs when two formerly disparate networks of ideas are connected using a bridge.
So to compose with all the sounds on Freesound, all I needed to do was to organize sounds into networks, then find new ways to connect them. But how could I organize sounds into networks, and what would a new connection look like?
The wordnik API
I realized that I could make lexical networks of sounds. The tags on sounds on Freesound give us a way to connect sounds. For instance, we could find all sounds that have the tag "scream" and form them into one network.
To make creative jumps, I had to bring in a new data source. After all, the sounds that share a tag are already connected.
That's when I incorporated The Wordnik API. Wordnik is an incredibly complete dictionary, thesaurus, and encyclopedia all wrapped into one. And best of all, they expost it using a fast and affordable API.
Composing with all sound using Freesound, a network model of creativity, and lexical relationships
So the final algorithm looks something like this, although there are many ways to vary it.
- Start with a search term provided by a user
- Build a network of sounds with that search term
- Use Wordnik to find related words
- Build a network around the related words
- Connect the original network to the new one
- Repeat
To sonify the resulting networks, I used a simple model of artificial intelligence that is sort of like a cellular automaton. I released a swarm of simple automata on the network and turned on sounds whenever the number of bots on a sound reached a critical mass.
Results
Here are the results of my dissertation. You can read a paper I presented at a conference, and the dissertation itself. Then you can listen to the music written by this program. It's not Beethoven, but I guarantee that you will find it interesting.
Composing with All Sound Using the FreeSound and Wordnik APIs
Method for Simulating Creativity to Generate Sound Collages from Documents on the Web
Disconnected: Algorithmic music composed using all sounds and a network model of creativity
How to host a static site on AWS
In this post, I'm going to show you how you can put up a simple html webpage on AWS using resources that are usually free. It's probably the cheapest way to set up a website.
Prerequisites
There are a few things you need to do ahead of time.
- Sign up for AWS
- Create a webpage
For the first one, head on over to https://aws.amazon.com/console/ and create a new account.
For the second one, you will probably need to learn a little coding to make it happen, but I'll give you something simple to start with.
<!DOCTYPE html>
<html>
<head>
<title>My Webpage</title>
</head>
<body>
<main>
<h1>My Webpage</h1>
<p>
This is a simple html webpage hosted on AWS S3 and AWS Cloudfront.
</p>
<p>
This was built based on <a href="https://evanxmerz.com/post/how-to-host-a-static-site-on-aws" target="_blank">a tutorial at evanxmerz.com/post/how-to-host-a-static-site-on-aws</a>.
</p>
</main>
</body>
</html>
There are some optional prerequisites that I'm intentionally omitting here. If you want to host your site on your own domain, then you'll need to purchase that domain from a domain registrar, such as AWS Route 53. Then you would also need to get an SSL certificate from a certificate authority such as AWS Certificate Manager.
Create a public S3 bucket
For your site to exist on the internet, it must be served by a computer connected to the internet. To make this possible, we need to upload your html file to "the cloud" somehow. Don't worry, "the cloud" is simply a marketing term for a web server. In this tutorial, our cloud storage service is AWS S3.
First you need to create a bucket.
- Browse to the AWS S3 service
- Click "Create Bucket". A bucket is a container for files. You might use a different bucket for each of your websites.
- Enter a name for your bucket. I name my static site buckets after the site they represent. So empressblog.org is in a bucket called empressblog-org. I named my bucket for this tutorial "static-site-tutorial-1-evanxmerz-com" because I am going to connect it to static-site-tutrial-1.evanxmerz.com.
- Next, select the AWS region for your bucket. The region is not really important, but you must write down what you select, because you will need it later. I selected "us-west-1".
- Under "Object Ownership" select "ACLs enabled". This will make it easier for us to make this bucket public.
- Under "Block Public Access settings for this bucket", unselect "Block all public access", then click the checkbox to acknowledge that you are making the contents of this bucket public.
- Then scroll to the bottom and click "Create Bucket".
Next, you need to allow hosting a public website from your bucket.
- Locate your bucket in the bucket list in S3, then click it.
- You should now be looking at the details for your bucket. Click the "Properties" tab, then scroll down ti "Block all public access" and click "Edit".
- Click "Enable", then under "Index document", enter "index.html". Then click "Save changes".
- Click the "Permissions" tab for your bucket. Scroll down to "Access control list (ACL)" and click "Edit". Next to "Everyone (public access)", click the box that says "Read". Then click the box that says "I understand the effects of these changes on my objects and buckets" and click "Save changes".
Create your index page
This step assumes that you have already created an html page that you want to make public. You can also upload other pages, css files, and javascript, using this same procedure.
- Find the "Objects" tab for your bucket on S3.
- Click "Upload".
- Click "Add files".
- Browse to your index.html
- Scroll down to the "Permissions" accordion and expand it.
- Click "Grant public-read access" and the checkbox to say that you understand the risk.
- Then click "Upload".
Now your page is on the internet. You can go to the objects tab for your bucket, then click on your file. That should display a link called "Object URL". If you click that link, then you should see your page. My link is https://static-site-tutorial-1-evanxmerz-com.s3.us-west-1.amazonaws.com/index.html.
Why to make a CloudFront distribution
Now you have seen your file on the web. Isn't that enough? Isn't this tutorial over? No. There are several problems with using S3 alone.
- Your files will be served from S3, which is not optimized for worldwide distribution. Your files have a home on a server in the region you selected. If that region is in the US, then people in Asia are going to have a much slower download experience.
- Your site will not work if there are errors. Your index page works fine, but what happens if you entered an incorrect link, and someone ends up at indec.html? You will get a nasty error message from AWS, rather than being redirected to a page on your site.
The final problem is the URL. This can be solved by setting up a domain in Route 53, but it's much wiser to set up a CloudFront distribution, then connect your domain to that.
Set up a CloudFront distribution
AWS CloudFront is a Ccontent Distribution Network (CDN). A CDN is a network of servers all over the world that are close to where people live. So people in Asia will be served by a copy of your page on a server in Asia.
- Find the CloudFront service on AWS.
- Click "Create distribution".
- In the "Origin domain" field you must enter your S3 bucket's Static Website Hosting Endpoint as the CloudFront origin. You can find this on the "Properties" tab of your S3 bucket. So for me, that was "static-site-tutorial-1-evanxmerz-com.s3-website-us-west-1.amazonaws.com".
- Then scroll all the way down to "Alternate domain name (CNAME)". This is where you would enter a custom domain that you purchased from AWS Route 53 or another registrar. For instance, if you want to set up your site on mystore.com, then you would enter "*.mystore.com" and "mystore.com" as custom domains. I entered "static-site-tutorial-1.evanxmerz.com" as my custom domain because that's where I'm putting up this tutorial.
- Then go to the "Custom SSL certificate" field. If you do not have your own domain, then you can ignore this. But if you have your own domain, then you will need your own SSL certificate. The SSL certificate is what enables private web browsing using the "https" protocol. Go set up a certificate using AWS Certificate Manager before setting up CloudFront if you want to use a custom domain.
- Finally, click "Create distribution"
Then you need to modify the distribution to act more like a normal web server. So we will redirect users to index.html if they request an invalid url.
- Find your distribution on CloudFron and click it.
- Click the "Error pages" tab.
- Click "Create custom error response".
- Under "HTTP error code" select "400: Bad Request".
- Under "Customize error response" click "Yes".
- Under "Response page path" enter "/index.html".
- Under "HTTP Response code" select "200: OK".
- Click "Create custom error response".
- Repeat these steps for 403 and 404 errors.
Then, if you have a custom domain, you need to go to AWS Route 53 and enable it.
- Go to Route 53 and select the hosted zone for your domain.
- Click "Create record".
- Create an A record for your domain or subdomain.
- Under "Route traffic to" click "Alias".
- Click "Alias to CloudFront distribution" and select your distribution.
- Click "Create records".
Now if you visit your custom domain, you should see your page. Here's mine: https://static-site-tutorial-1.evanxmerz.com/
Congratulations!
Congratulations! You've erected your first website using AWS S3 and AWS CLoudFront! Let's review the basic architecture here.
- Files are stored in a cloud file system called AWS S3.
- Files are served by a Content Distribution Network (CDN) called AWS CloudFront.
- Optionally, domains are routed to your CloudFront distribution by AWS Route 53.
- Optionally, CloudFront uses an SSL certificate from AWS Certificate Manager.
This is about the simplest and cheapest architecture for hosting a fast static site on the internet.
It's important to note that this is also the simplest way for hosting ANY static site. If you generated a static site using React, Gatsby, or Next, then you could host them in the same way.
IT's also important to note that this architecture fails as soon as you need to make decisions server side. This architecture works fine for websites that are frontend only websites, where you don't interact with private data. Once you need private data storage, an API, or custom logic on a page, then you will need a server in some variety. There are multiple solutions in that case, from the so-called "serverless" AWS Lambda, or the more old-fashioned AWS EC2, which is simply a server farm.
But you are now ready to start exploring those more complex options.
How to conquer Google Core Web Vitals
In 2021, Google finally launched their long awaited Core Web Vitals. Core Web Vitals are the measurements that Google uses to measure the user experience (UX) on your website. These include metrics tracking things like how long the page takes to load images, how long it takes to become interactive, and how much it jitters around while loading.
Core Web Vitals are important to all online businesses because they influence where your site appears on Google when users are searching for content related to your site. They are far from the only consideration when it comes to search engine optimization (SEO), but they are increasingly important.
In this article, I'm going to give you a high level overview of how you can get green scores from Google's Core Web Vitals. We're not going to dig into implementation level details, yet, but we will look at the general strategies necessary to optimize the scores from Google.
This article is informed by my experience optimizing Core Web Vitals scores for the websites I manage and develop professionally. It's also informed by my study and experimentation on my own websites and projects.
Prepare for a long battle
Before we begin, I want to set your expectations properly. On any reasonably large website, Core Web Vitals scores cannot be fixed in a night, or a weekend, and maybe not even in a month. It will likely take effort each week, and in each development project, to ensure that your scores get to the highest level and stay there.
This is the same expectation we should have for any SEO efforts. It takes time to get things right in the code, the architecture, and in the content. All three of those things must work together to put your site on the first page of Google search results for high volume search terms.
In my experience, a successful SEO operation requires coordination of web developers, SEO experts, writers, and whoever manages the infrastructure of your website.
1. Understand Core Web Vitals
The first step to conquering Core Web Vitals is to understand what they are measuring. So get access to Google Search Console, and start looking at some reports. Use PageSpeed Insights to start looking at the values that Google is actually calculating for your site.
Don't be scared if your scores are really low at first glance. Most sites on the internet get very low scores because Google's standards are incredibly high.
You should see some aronyms that represent how your site did on key metrics. There are four metrics that are used to calculate the Core Web Vitals score for a page.
- LCP = Largest Contentful Paint.
- The LCP of a page is how long it takes for the largest thing to be shown on screen. Typically this is the banner image on a page, but PageSpeed Insights can sometimes get confused if you use header tags in the wrong places.
- FCP = First Contentful Paint.
- The FCP of a page is how long it takes for anything to appear on screen.
- CLS = Cumulative Layout Shift.
- The CLS is a measure of how much elements on the page jitter around while the page is loading.
- FID = First Input Delay.
- FID is how long it takes for the page to respond to a click.
2. Make the right architecture decisions when starting a new site
Most people can't go back in time and choose a different platform for their site. Most people are stuck with the architectural decisions made at the beginning of the project. If you're in that category, then don't fret. The next section will talk about how you can maximize existing infrastructural technologies to get the maximum performance out of your website.
On the other hand, if you are starting from scratch then you need to critically consider your initial architectural decisions. It's important to understand how those architectural decisions will impact your Core Web Vitals score, and your search ranking on Google.
Client side rendering vs. Sever side rendering
The most important decision you have to make is whether you will serve a site that is fully rendered client side or a site that renders pages on the server side.
I want to encourage you to choose a site architecture that is fully client side rendering, because client side rendering provides huge benefits when it comes to both scaling and SEO, and the drawbacks previously associated with client side rendering are no longer relevant.
Imagine the typical render process for a webpage. This process can be broken down into many steps, but it's important to realize that there are 3 phases.
- The server fetches data from a data store.
- The server renders the page.
- The client renders the page.
Steps 1 and 3 are unavoidable. If you have a backend of any complexity, then you must have a data store. If you want users to see your website, then their web browsers must render it.
But what about step 2? Is that step necessary? No. Even worse is the fact that it's often the most time-consuming step.
But if you build your site as a client-side js/html/css bundle that uses an API on the backend, then you can cut out the server side rendering entirely. If you want to maximize your site's Core Web Vitals scores, and hence your SEO, then you should cut out the server side rendering step if possible. Adding that step is inherently slower than strategies that eliminate it.
This means that platforms that are used for server side rendering are probably something to avoid in 2022 and beyond. So Ruby on Rails is probably a bad choice, unless you're only using it to build an API. Similarly, Next.js is probably a bad choice, unless you are only using it to generate a static site.
You should also look at benchmarks for various website architectures. You will see that vanilla React/Vue/Javascript sites generally outperform server side rendered sites by a wide margin.
But isn't a client side rendered site bad for SEO?
If someone hasn't done much SEO lately, then they may come up with this counterargument, that using client side rendering is bad for SEO. Up until around 2017, Google didn't run the javascript on web pages it was scanning. If the javascript wasn't executed, then a purely client side rendered page would show up as an empty page and never rank on Google. We had to come up with lots of workarounds to avoid that fate.
Since 2017, however, Google does run the javascript on the page. So having a purely client-side rendered site is perfectly fine.
Server side rendered sites are also harder to scale
I've scaled several websites in my career, so I can tell you that it's much more difficult to scale a server side rendered website than one that isn't server side rendered. To scale on the server side requires a complex balance of caching, parallelization, and optimization that is entirely unnecessary if your site is rendered client side.
It's true that you would still have to scale the API on the server side, but that's true in both cases.
3. Maximize mature web technologies
But what if you can't change the fundamental architecture of your website? Are you stuck? No. There are a number of web technologies that exist to mitigate the performance impact of various architectural decisions.
- Use a Content Distribution Network, especially for images. Take a look at services like ImgIX, which can optimize your images and play the role of a CDN. Other commonly used CDNs include CloudFront and CloudFlare.
- Optimize your cache layer. Try to hit the database as rarely as possible. The API that backs your site will get bottlenecked by something. Quite frequently, that something is your database. Use a cache layer such as REDIS to eliminate database queries where possible.
- Use geolocated servers. The CDN should take care of serving content from locations near your users, however, if requests are ever getting back to your web servers, make sure you have servers near your users.
- Enable automatic horizontal scaling of your server. You need to be able to add more servers during high traffic times, such as the holidays. To do so requires parallelizing your server code, and handling session across multiple servers. There are many ways to mitigate those problems, such as the sticky sessions feature offered by Heroku. You need to be able to scale horizontally at least on your API, if not on the servers that serve your site.
4. Monitor the results
Finally, you need to monitor Google Search Console every day. Google Search Console is one of the first things I check every time I open my work computer to begin my day. Make this a habit, because you need to be aware of the trends Google is seeing in the user experience on your site. If there is a sudden issue, or things are trending the wrong way, you need to proactively address the problem.
You should also use other site monitoring tools. Google's scores are a running 28 day average. You need a response time faster than 28 days. So I encourage you to set up a monitoring service such as Pingdom or Status Cake, which will give you real time monitoring of website performance.
Finally, you should run your own spot checks. Even if everything looks okay at a glance, you should regularly check the scores on your most important pages. Run PageSpeed Insights or Web Page Test on your homepage and landing pages regularly. Make sure to schedule time to take care of the issues that arise.
Conclusion
To conquer Google Core Web Vitals requires coordination of the effort of many people over weeks or months of time, but with a rigorous approach you can get to the first or second page of Google search results for high volume search terms. Make sure to spend time understanding Google Core Web Vitals. Ensure that the critical architectural decisions at the beginning of a project are in line with your long term goals. Maximize your use of mature web technologies to mitigate architectural issues. And monitor your site's performance every day.
With these strategies you can take your site to the next level.
What are common mobile usability issues and how can you solve them?
In this article, I'm going to show you some common mobile usability issues reported by Google, and show you how to fix them using basic css.
Mobile usability issues in Google Search Console
A few days ago I received this rather alarming email from Google.
I was surprised to see this, because, although my blog isn't the most beautiful blog on the web, I pride myself on the clarity for reading on any device. So I logged into Google Search Console and found the "Mobile Usability" link on the left side to get more information.
Search console gave me the same three issues.
- Viewport not set
- Text too small to read
- Clickable elements too close together
Search console said they were all coming from https://evanxmerz.com/soundsynthjava/Sound_Synth_Java.html. This url points at the file for my ebook Sound Synthesis in Java. This ebook was written using markup compatible with early generations of Kindle devices, so it's no surprise that Google doesn't like it.
When I opened the book on a simulated mobile device, I could see that the reported mobile usability issues were apparent.
Let's go through each of the issues and fix them one by one. This will require some elementary CSS and HTML, and I hope it shows how even basic coding skills can be valuable.
Viewport not set
Viewport is a meta tag that tells mobile devices how to interpret the page on a device. We need to provide it to give web browsers a basis for showing a beautiful page to viewers. In practical terms, this means that without a viewport meta tag, the fonts on the page will look very small to viewers on mobile devices because their browsers don't know how to scale the page.
To solve this issue, add this line to the html head section.
<meta name="viewport" content="width=device-width, initial-scale=1">
This may also solve the remaining issues, but we'll add some css for them just to be sure.
Add a CSS file
The following two fixes require us to use some css, so we will need to add a CSS file to our webpage.
Early kindles didn't support much css, so it was better to just format your books with vanilla html and let the Kindle apply formatting. That limitation doesn't apply to the web, though, so I created a file called styles.css in the same directory as my html file, and added this line within the head section of my html page. This tells the page to pull style information from the file called "styles.css".
<link rel="stylesheet" href="styles.css">
Next we need to add some style rules to fix the remaining mobile usability issues.
Text too small to read
Making the font larger is easy. We could do it for only mobile devices, but I think increasing the font size slightly will help all readers. The default font size is "1rem" so let's bump it up to 1.1 by adding the following code to our css file.
body {
font-size: 1.1rem;
}
Clickable elements too close together
The only clickable elements in this document are links. So this means that the text is too small, and the lines are too close together. The previous two fixes might also address this issue, but just in case they don't, let's make the lines taller by adding another line to our body css.
body {
font-size: 1.1rem;
line-height: 1.5;
}
The default line-height is 1. By moving it to 1.5, we are making lines 50% taller. This should make it easier for users to click the correct link.
Putting it all together
In addition to solving the mobile usability issues from Google, I wanted to ensure a good reading experience on all devices, so here's my final set of rules for the body tag in my css file.
At the bottom I added rules for max-width, margin, and padding. The max-width rule is so that readers with wide monitors don't end up with lines of text going all the way across their screens. The margin rule is an old-fashioned way of horizontally centering an element. The padding rule tells the browser to leave a little space above, below, and beside the body element.
body {
font-size: 1.1rem;
line-height: 1.5;
max-width: 800px;
margin: 0 auto;
padding: 10px;
}
When I opened up the result in a simulated mobile device, I could see that the issues were fixed.
How to make Dropbox ignore the node_modules folder
In this article I'm going to show you how to get Dropbox to ignore the node_modules folder.
Why code in a Dropbox folder?
I recently bought a new computer and experienced the pain of having to sync around 300Gb of files from one Windows machine to another. This was primarily due to the in-progress music projects on my computer.
So I decided that I would just work within my Dropbox folder on my new computer. In the past, I've spoken to colleagues who did this. They were mostly artists, and didn't use git.
What I didn't consider at the time, was that this would mean there would be multiple syncing mechanisms for my code. I would be sending the files both to Dropbox and Git.
Even worse, Dropbox locks files while it is working. So if you are running something like "npm install" or "npm update", those processes can fail because they can't work on a file at the same time as Dropbox.
I began experiencing these problems almost immediately, so I either had to find a way to ignore the node_modules folder in Dropbox or stop working on code within the Dropbox folder.
Dropbox ignore syntax
Fortunately, Dropbox gave us a way to ignore files or folders a few years ago. So for Windows users, the following command will ignore your node_modules folder. Please note that this needs to be done for every project, and that the command needs to be completed with your actual directory structure.
Set-Content -Path 'C:\Users\yourname\Dropbox\yourproject\node_modules' -Stream com.dropbox.ignored -Value 1
How to optimize largest contentful paint (LCP) on client side
In this article I'm going to show you some strategies I've used to optimize websites on the client side for largest contentful paint (lcp) and first contentful paint (fcp).
What is largest contentful paint and why does it matter?
Largest contentful paint (LCP) is a measure of how long from initial request it takes for your site to render the largest thing on screen. Usually it measures the time it takes your largest image to appear on screen.
First contentful paint (FCP) is a measure of how long from initial request it takes for your site to render anything on screen. In most cases, optimizing LCP will also optimize FCP, so this is the last time I'll mention FCP in this article.
Google Core Web Vitals considers 2.5 seconds on mobile to be a good LCP when loading the site at 4g speeds. This is an extremely high bar to reach, and most sites won't come close without addressing it directly.
It's also important to note that reaching an LCP of 2.5 seconds once is not sufficient to pass. Your pages must achieve an average LCP of under 2.5 seconds over a period of 28 days in order to pass. This means that putting off work on LCP will only be more painful in the future. You need to act now to move your LCP in the right direction.
Why not use server side rendering?
When searching for ways to optimize LCP, I came across various sites that suggested server side rendering. They suggest that rendering the page server side and delivering it fully rendered client side would be the fastest. I know from experience that this is wrong for several reasons.
First, you still need to render on client side even if you deliver a flat html/js/css page. The client side still needs to extract and compile the page, and that takes the bulk of rendering time for modern, js-heavy webpages.
Second, rendering server side can only possibly be faster if your site isn't scaling. Yes, when you have a small number of simultaneous users, it's much faster to render on your server than on an old android. Once you hit hundreds or thousands of simultaneous users that math quickly flips, and it's much faster to render on thousands of client machines, no matter how old they are.
Why not use service workers?
Another suggestion I see is to use service workers to preload content. Please keep in mind that Google only measures the first load of a page. It does not measure subsequent loads. So any technique that improves subsequent loads is irrelevant to Google Core Web Vitals. Yes, this is incredibly frustrating, because frameworks like Next.js give you preloading out of the box.
Optimize images using modern web formats and a CDN
The most important thing you can do to achieve a lower LCP is to optimize the delivery of your images. Unless you use very few images, as I do on this blog, then images are the largest part of the payload for your webpages. They are typically around 10 to 100 times larger than all other assets combined. So optimizing your images should be your number one concern.
First, you need to be using modern web image formats. This means using lightweight formats such as webp, rather than heavier formats such as png.
Second, you need to deliver images from a content distribution network (CDN). Delivering images from edge locations near your users is an absolute must.
Third, you need to show images properly scaled for a user's device. This means requesting images at the specific resolution that will be displayed for a user, rather than loading a larger image and scaling it down with css.
Finally, Google seems to prefer progressive images, which give the user an image experience such as a blurred image before the full image has loaded into memory. There are many robust packages on the web for delivering progressive images.
I suggest you consider ImgIX for optimizing your images. ImgIX is both an image processor and a CDN. With open source components that work with various CMSs, and in various environments, ImgIX is a one stop shop that will quickly solve your image delivery issues. I've used it at two scaling websites, and in both cases it has been extremely impactful.
Deliver the smallest amount of data to each page
After you optimize your images, the next thing to consider is how much data you are sending to the client. You need to send the smallest amount of data that is necessary to render the page. This is typically an issue on list pages.
If you're using out-of-the-box CRUD APIs built in Ruby on Rails, or many other frameworks, then you typically have one template for rendering one type of thing. You might have a product template that renders all the information needed about a product. Then that same product template is used on product detail pages, and on list pages. The problem with that is that much less information is needed on the list pages. So it's imperative that you split your templates into light and heavy templates, then differentiate which are used in which places.
This is more of a backend change than a frontend change, but optimizing frontend performance requires the cooperation of an entire team.
Deliver static assets using a CDN
After putting our images through ImgIX, we stopped worrying about CDNs. We thought that because images were so much larger than the static assets, it wouldn't make much difference to serve static assets from our servers rather than a CDN.
This is true, if you are just beginning to optimize your frontend performance. Putting static assets on a CDN won't lead to a tremendous drop in LCP.
However, once you are trying to get your page load time down to the absolute minimum, every little bit counts. We saved an average of around two tenths of a second on our pages when we put our static assets on a CDN, and two tenths of a second is not nothing.
Another great thing about putting your static assets on a CDN is that it typically requires no code changes. It's simply a matter of integrating the CDN into your continuous integration.
Eliminate third party javascript
Unfortunately, third party javascript libraries are frequently sources of a significant amount of load time. Some third party javascript is not minimized, some pulls more javascript from slow third party servers, and some uses old fashioned techniques such as document.write.
To continue optimizing our load time we had to audit the third party javascript loaded on each page. We made a list of what was loaded where, then went around to each department and asked how they were using each package.
We initially found 19 different trackers on our site. When we spoke with each department we found that 6 of them weren't even being used any more, and 2 more were only lightly used.
So we trimmed down to 11 third party javascript libraries then set that as a hard limit. From then on, whenever anyone asked to add a third party library, they had to suggest one they were willing to remove. This was a necessary step to meet the aggressive performance demands required by Google.
Optimize your bundle size
The final thing to do to optimize your client side load time is to optimize your bundle size. When we talk about bundle size, we're talking about the amount of static assets delivered on your pages. This includes javascript, html, css, and more. Typically, extracting and compiling javascript is what takes the bulk of the time, so that's what you should focus on.
Use code splitting
Code splitting means that your app generates multiple bundles that are potentially different for each page. This is necessary in order to deliver the smallest amount required for a given page. Most modern website transpilers like WebPack will do this automatically.
Forget import *
Stop using "import *" entirely. You should only ever import the methods you are using. When you "import *" you import every bit of code in that module as well as every bit of code that it relies on. In most circumstances, you only need a fraction of that.
It's true, that a feature called tree shaking is able to eliminate some of the cruft in scenarios where you're importing more than you need, but it's sometimes tough to figure out where the tree shaking is working and where it's failing. To do so, you need to run bundle analysis, and comb through it carefully.
It's much easier to simply stop using "import *".
Use composition wisely
I almost named this section "Forget the factory pattern", because the factory pattern creates situations very similar to "import *". In the factory pattern, a method is called that returns an object with all the methods needed to fulfill a responsibility or interface. What I see most often, is a misapplication of the factory pattern whereby programmers are dumping a whole bunch of methods into a pseudo-module then using only one or two methods.
// don't do this
const createDateHelpers = () => {
const formatDate = () => {...};
const dateToUtc = () => {...};
return {
formatDate,
dateToUtc,
}
}
You can see that if you want to call "formatDate", then you need to run "createDateHelpers().formatDate()". This is essentially the same as importing every method in the date helpers module, and again, you are importing all their dependencies as well.
This is where composition can be applied to make an object that gives you the full object when needed, but also allows you to export methods individually.
// use composition
export const formatDate = () => {...};
export const dateToUtc = () => {...};
export default const createDateHelpers = () => {
return {
formatDate,
dateToUtc,
}
};
Render a simplified static page
It's important to note that optimizing your existing pages isn't the only strategy available. Amazon's website crushes Google's Core Web Vitals, even though it isn't very fast. It does this by rendering a simplified static template for the first load, then filling it in with the correct content. So if you visit Amazon, you may see some evergreen content flash up on the page before content specific to you loads in.
That's a fine way to pass Google's Core Web Vitals, but it isn't optimizing the performance of your page. That's tailoring your page to meet a specific test. It's not cheating, but it's not necessarily honoring the intention of Google's UX metrics.
Conclusion
There are two basic categories of changes that are necessary to optimize the client side of a website for largest contentful paint: optimizing delivery of images, and optimizing delivery of code. In this article I've listed several strategies I've used in the past to address both of these categories. These strategies include using progressive images, using CDNs, and delivering as little data and code as is necessary to render a page.
How to measure the performance of a webpage
Measuring the performance of a webpage is an extremely complex topic that could fill a book. In this article I'm going to introduce some of the commonly used tools for measuring web performance, and show you why you need to take multiple approaches to get a complete picture of the performance of any webpage.
Why measure webpage performance?
Why measure webpage performance? Performance is so critical to the modern web that this question seems almost comical. Here are just a few reasons why you should be measuring and monitoring webpage performance.
- User experience impacts Google search rankings via Core Web Vitals.
- Page load time correlates with conversion rate and inversely correlates with bounce rate.
- Pages must be performant to be accessible to the widest possible audience.
Four approaches to measuring performance
No single tool is going to give you a comprehensive perspective on webpage performance. You must combine multiple approaches to really understand how a page performs. You must look at multiple browsing patterns, multuple devices, multiple times of day, and multiple locations.
In this article, I'm going to talk about four different approaches to measuring the performance of a page and recommend some tools for each of them. You must use at least one from each category to get a complete picture of the performance of a webpage.
The four approaches I'm going to talk about are...
- One time assessments
- Live monitoring and alerts
- Full stack observability
- Subjective user tests
Why do we need multiple perspectives to measure performance?
People who are new to monitoring web performance often make mistakes when assessing web performance. They pull up a website on their phone, count off seconds in their head, then angrily email the developers complaining that the website takes 10 seconds to load.
The actual performance experienced by users of your website is an important perspective, but there are dozens of things that can impact the performance of a single page load. Maybe the page wasn't in the page cache. Maybe the site was in the middle of a deploy. Maybe hackers are attacking network infrastructure. Maybe local weather is impacting an ISP's network infrastructure.
You can't generalize from a single page load to the performance of that page. I've seen many people fall into this trap, from executives to marketing team members. It makes everyone look bad. The website looks bad because of the slow page load. The developers look bad because of the poor user experience. The person who called it out looks bad because it looks like they don't know what they are doing. The data analysts look bad because they aren't exposing visualizations of performance that other team members can use.
So read this article and fire up some of the tools before sending out that angry email.
Please note that the tools listed here are just the tools that I have found to be effective in my career. There are many other options that may be omitted simply because I've never used them.
One time assessments
One time assessments are the most commonly used tools. One time assessments can be run against a production webpage at any time to get a perspective of the performance at that moment. One thing that's nice about these tools is that they can be used effectively without paying anything.
PROS:
- Easy to use
- Easy to quantify
- Fast
- Reliable
- Affordable
CONS:
- May lack perspective over time
- Lacks perspective on actual browsing patterns, including subsequent page loads
- Lacks perspective on other locations
- May lack information on the source of an issue
TOOLS:
Live monitoring and alerts
The performance of a webpage can degrade very quickly. It can degrade due to poor network conditions, an influx of bot visitors from a new ad, a influx of legitimate visitors during peak hours, or from the deploy of a non-performant feature.
When the performance does degrade, you need to know immediately, so you can either roll back the deploy, or investigate the other factors that may be slowing down the site.
Notice that price isn't listed as a benefit (pro) or drawback (con) of live monitoring tools. You generally need to pay something to use these tools effectively, but usually that price is less than $100 a month, even on large sites.
PROS:
- Real-time notifications of performance changes
- Easy to quantify
- Can be configured to request from other locations
CONS:
- Limited information
- Lacks perspective over time
- May lack information on the source of an issue
- Fragile configuration can lead to false positives
TOOLS:
Full stack observability
Some tools offer insights into the full page lifecycle over time. These tools are constantly ingesting data from your website and compiling that data into configurable visualizations. They look at page load data on the client side and server side using highly granular measurements such as database transaction time, and the number of http requests.
If you wanted a single source of information to measure performance on your site, then these tools are the only option. They can provide one time assessments, monitoring, and backend insights.
One big problem with these tools is that they are quite complex. To use these tools effectively, you need developers to help set them up, and you need data analysts to extract the important information exposed by these tools.
PROS:
- Includes detailed breakdowns that can help identify the source of performance issues
- Includes data over time
- Highly granular data
CONS:
- Difficult to use
- Expensive
- Requires developer setup and configuration
TOOLS:
Qualitative user tests
The final approach for measuring the performance of a webpage is subjective. In other words, you must actually look at how your site performs when real people are using it. This can be as simple as opening a website on all your devices and trying to browse like a normal user, or you can set up time to interview real users and gather qualitative information about their experience.
I once worked at a company that required developers to attend in-person user tests every two weeks. This allowed every developer to see how users actually browsed and experienced their work. This may be overkill for most companies, but it's a perspective that can't be ignored.
PROS:
- No additional tools are necessary
- Exposes actual, real world user experience
- Exposes issues raised from real browsing patterns including subsequent page loads
CONS:
- It's easy to prematurely generalize from a small number of cases
- Can be expensive and difficult to do well
- Difficult to quantify
- Not timely
TOOLS:
- Web browsers on multiple devices
- Google Analytics
- User testing services
Conclusion
In this article, I introduced four different ways to measure the performance of a webpage. Each of them is necessary to get a full understanding of the performance of a page. I also introduced some of my favorite tools that can be used for each approach.
The four approaches are...
- One time assessments
- Live monitoring and alerts
- Full stack observability
- Subjective user tests
I hope that this prepares you to start wading into the complex world of measuring website performance.
Evan's React Interview Cheat Sheet
In this article I'm going to list some of the things that are useful in React coding interviews, but are easily forgotten or overlooked. Most of the techniques here are useful for solving problems that often arise in coding interviews.
Table of contents
- Imitating componentDidMount using hooks
- Using the previous state in the useState hook
- Using useRef to refer to an element
- Custom usePrevious hook to simplify a common useRef use case
- Vanilla js debounce method
- Use useContext to avoid prop-drilling
Imitating componentDidMount using React hooks
The useEffect hook is one of the more popular additions to the React hooks API. The problem with it is that it replaces a handful of critical lifecycle methods that were much easier to understand.
It's much easier to understand the meaning of "componentDidMount" than "useEffect({}, [])", especially when useEffect replaces componentDidMount, componentDidUpdate, and componentWillUnmount.
The most common use of the useEffect hook in interviews is to replace componentDidMount. The componentDidMount method is often used for loading whatever data is needed by the component. In fact, the reason it's called useEffect is because you are using a side effect.
The default behavior of useEffect is to run after ever render, but it can also be run conditionally using a second argument that triggers the effect when it changes.
In this example, we use the useEffect hook to load data from the omdb api.
// fetch movies. Notice the use of async
const fetchMovies = (newSearchTerm = searchTerm) => {
// See http://www.omdbapi.com/
fetch(`http://www.omdbapi.com/?apikey=${apikey}&s=${newSearchTerm}&page=${pageNumber}`).then(async (response) => {
const responseJson = await response.json();
// if this is a new search term, then replace the movies
if(newSearchTerm != previousSearchTerm) {
setMovies(responseJson.Search);
} else {
// if the search term is the same, then append the new page to the end of movies
setMovies([...movies, ...responseJson.Search]);
}
}).catch((error) => {
console.error(error);
});
};
// imitate componentDidMount
useEffect(fetchMovies, []);
Note that this syntax for using asynchronous code in useEffect is the suggested way to do so.
Using the previous state in the useState hook
The useState hook is probably the easiest and most natural hook, but it does obscure one common use case. For instance, do you know how to use the previous state in the useState hook?
It turns out that you can pass a function into the useState set method and that function can take the previous state as an argument.
Here's a simple counter example.
const [count, setCount] = useState({});
setCount(prevState => {
return prevState + 1;
});
Using useRef to refer to an element
The useRef hook is used to store any mutable value across calls to the component render method. The most common use is to use it to access an element in the DOM.
Initialize the reference using the useRef hook.
// use useRef hook to keep track of a specific element
const movieContainerRef = useRef();
Then attach it to an element in the render return.
<div className={MovieListStyles.movieContainer} ref={movieContainerRef}>
{movies && movies.length > 0 &&
movies.map(MovieItem)
}
</div>
Then you can use the .current property to access the current DOM element for that div, and attach listeners or do anything else you need to do with a div.
// set up a scroll handler
useEffect(() => {
const handleScroll = debounce(() => {
const scrollTop = movieContainerRef.current.scrollTop;
const scrollHeight = movieContainerRef.current.scrollHeight;
// do something with the scrolling properties here...
}, 150);
// add the handler to the movie container
movieContainerRef.current.addEventListener("scroll", handleScroll, { passive: true });
// remove the handler from the movie container
return () => movieContainerRef.current.removeEventListener("scroll", handleScroll);
}, []);
Custom usePrevious hook to simplify a common useRef use case
The useRef hook can be used to store any mutable value. So it's a great choice when you want to look at the previous value in a state variable. Unfortunately, the logic to do so is somewhat tortuous and can get repetitive. I prefer to use a custom usePrevious hook from usehooks.com.
import {useEffect, useRef} from 'react';
// See https://usehooks.com/usePrevious/
function usePrevious(value) {
// The ref object is a generic container whose current property is mutable ...
// ... and can hold any value, similar to an instance property on a class
const ref = useRef();
// Store current value in ref
useEffect(() => {
ref.current = value;
}, [value]); // Only re-run if value changes
// Return previous value (happens before update in useEffect above)
return ref.current;
}
export default usePrevious;
Using it is as simple as one extra line when setting up a functional component.
// use the useState hook to store the search term
const [searchTerm, setSearchTerm] = useState('orange');
// use custom usePrevious hook
const previousSearchTerm = usePrevious(searchTerm);
Vanilla js debounce method
Okay, this next one has nothing to do with React, except for the fact that it's a commonly needed helper method. Yes, I'm talking about "debounce". If you want to reduce the jittery quality of a user interface, but you still want to respond to actions by the user, then it's important to throttle the rate of events your code responds to. Debounce is the name of a method for doing this from the lodash library.
The debounce method waits a preset interval until after the last call to debounce to call a callback method. Effectively, it waits until it stops receiving events to call the callback. This is commonly needed when responding to scroll or mouse events.
The problem is that you don't want to install lodash in a coding interview just to use one method. So here's a vanilla javascript debounce method from Josh Comeau.
//
const debounce = (callback, wait) => {
let timeoutId = null;
return (...args) => {
window.clearTimeout(timeoutId);
timeoutId = window.setTimeout(() => {
callback.apply(null, args);
}, wait);
};
}
export default debounce;
Here's an example of how to use it to update a movie list when a new search term is entered.
// handle text input
const handleSearchChange = debounce((event) => {
setSearchTerm(event.target.value);
fetchMovies(event.target.value);
}, 150);
return (
<div className={MovieListStyles.movieList}>
<h2>Movie List</h2>
<div className={MovieListStyles.searchTermContainer}>
<label>
Search:
<input type="text" defaultValue={searchTerm} onChange={handleSearchChange} />
</label>
</div>
<div
className={MovieListStyles.movieContainer}
ref={movieContainerRef}
>
{movies && movies.length > 0 &&
movies.map(MovieItem)
}
</div>
</div>
);
Use useContext to avoid prop-drilling
The last thing an interviewer wants to see during a React coding interview is prop drilling. Prop drilling occurs when you need to pass a piece of data from one parent component, through several intervening components, to a child component. Prop drilling results in a bunch of repeated code where we are piping a variable through many unrelated components.
To avoid prop drilling, you should use the useContext hook.
The useContext hook is a React implementation of the provider pattern. The provider pattern is a way of providing system wide access to some resource.
It takes three code changes to implement the useContext hook. You've got to call createContext in the parent component that will maintain the data. Then you've got to wrap your app with a special tag.
export const DataContext = React.createContext()
function App() {
const data = { ... }
return (
<div>
<DataContext.Provider value={data}>
<SideBar />
<Content />
</DataContext.Provider>
</div>
)
}
Then you've got to import the context in the child component, and call useContext to get the current value.
import DataContext from '../app.js';
...
const { data } = React.useContext(DataContext);
What has made you stumble in React interviews?
What are some other common ways to make mistakes in React coding interviews? Send me your biggest React coding headaches on Twitter @EvanXMerz.
Why is software development so slow?
Why can't the developers move more quickly? Why does every project take so long? In this article I'm going to talk about what causes software development to slow down. This is based on my experience as a programmer and manager in Silicon Valley for over a decade, as well as conversations with other people in similar roles.
Is my team slow because...
First of all, I want to answer some questions that I've heard from managers.
- Is my team slow because we use PHP? No.
- Is my team slow because we use Java? No.
- Is my team slow because we use INSERT PRODUCT HERE? Probably not.
- Is my team slow because they don't understand web 3? No.
- Is my team slow because of a bad manager? Maybe, but if you've had multiple managers in there who haven't moved faster then this is unlikely.
- Is my team slow because they have or do not have college degrees? No.
- Is my team slow because we don't use agile? Probably not.
So then why is my development team so slow?
Now that we've gotten that out of the way, we can drill into what actually causes teams to move slowly. None of the reasons I've listed here are particularly technical. I tend to think that teams move slowly due to non-technical reasons.
They're estimating poorly
The most common reason why people think developers are moving slowly is that the estimates are poor. For one reason or another, developers are not spending the time, or creating the artifacts necessary to generate accurate estimates.
I once joined a small team where nobody ever wrote anything down. They would attend planning meetings where they would hash out the requirements, they would make up an estimate on the spot, then they would immediately break to start writing code. The stakeholders wanted to know why tasks never seemed to come in within the estimated time frame.
In my experience, it's necessary to generate multiple levels of plans to generate an accurate software development project estimate. You must begin with use-cases, then work through multiple levels of plans to reach units of work that can be estimated.
Here's how I break down a project to make an accurate estimate:
- Write all requirements down in plain language. This can be in the form of use cases, or user stories, or any top-level form that anyone can understand.
- Talk through what an acceptable solution would look like with other developers.
- Break that solution into individual tasks using a tool like JIRA.
- Estimate those tasks.
- Add a little extra room for unknowns, debugging, integration, and testing.
- Add up the final estimate.
There are multiple ways to execute each of these steps, but that's the general flow for all successful software development estimation that I've seen in my career.
Technical debt
Another big reason why a team might be moving slowly is tech debt. Tech debt happens to every project when the company tries to move quickly. It happens because solutions that can be implemented as fast as possible are usually not scalable to future use cases that the developers can't anticipate.
I once worked on a web development team that needed to make the list pages faster on an e-commerce website. The list pages were slow because the server was delivering a massive payload containing every product on the list page then filtering and sorting was done client side. That's easy to fix right? Just serve the first page of products, then serve the next when the user scrolls down. But to do that, we must add server side sorting and filtering because the list will be different for different combinations of sorts and filters. But to do that, we must have products stored in some sort of database, and this team was only using a key-value store at the time. That's how tech debt can be problematic. In that scenario, it would take months to solve that problem in a scalable way.
Ultimately, we found a different solution. We delivered the tiniest possible payload for each product, which achieved a similar end goal while side-stepping the tech debt. But that tech debt was still there, and would still be a problem for many related projects.
There is no silver bullet for solving tech debt. In my career, the best way I've found to deal with tech debt is to reserve one developer each sprint to iteratively move the project closer to what the team agrees is a more scalable solution.
Interpersonal conflict
The single biggest thing that slows down development teams is interpersonal and interdepartmental conflict. Maybe a manager is at odds with a team member, or maybe two team members see their roles very differently. If two people working on one project have different views about how to execute that project it can be absolutely crushing.
I think that proactively avoiding and addressing conflict is the single thing that can instantly help teams move faster. The most staggeringly successful group of techniques I've discovered for doing so is called motivational interviewing.
Motivational interviewing is a way of communicating with peers, stakeholders, and direct reports that allows two people to agree to be on the same team. It's a way of building real rapport with people that allows you to build successful working relationships over time. Specifically, it's a way of focusing a conversation on talk that meaningfully moves both parties toward a mutually acceptable solution.
I strongly recommend Motivational Interviewing for Leadership by Wilcox, Kersh, and Jenkins. To say that that book changed my life is an understatement.
Unclear instructions
The final category of issue that slows down development teams is unclear instructions.
I once worked for a person who was very fond of sending short project descriptions over email and expecting fast results. One email might read something like "I want to be selling in Europe in one month", another might say "I want to be able to hold auctions for some of our products."
It IS possible to work with such minimal instructions. It falls upon the development team to outline what they think the stakeholder wants, develop plans based on that outline, then execute based on those plans.
The problem with working from minimal instructions is that no two people see the same problem in exactly the same way. Say you are developing that auction software. Many questions immediately arise. Who can bid on the auctions? Is the auction for many products or just one? Does the product have selectable options, or are they preset? Are bids visible to other customers? How do we charge the user for the item if the final bid exceeds what is typically put on a credit card?
If the stakeholder isn't able to answer those questions early in the process, then it may go significantly astray. If the development team doesn't understand the instructions, then they may waste days or weeks pursuing irrelevant or untimely work. If the stakeholder is going to give such vague instructions then they must address questions in a timely manner or risk tasting time.
In conclusion
So why is your development team so slow? I've laid out the four common causes that I've seen in my career.
- Poor estimates
- Tech debt
- Interpersonal conflict
- Unclear instructions
If you have something else to add to this list, then let me know on Twitter @EvanXMerz.
How to generate and view a node bill of materials
In this post, I'm going to show you how you can generate and view a software bill of materials (sbom) for a node.js project.
Why generate a bill of materials?
With the increasing attacks on the software supply chain by malicious actors such as foreign states and crypto miners, it's important to understand exactly what packages are in your node project. It's also important to understand the vulnerabilities in each of those packages. The tools in this tutorial link out to Socket.dev, which analyzes npm packages to find potential vulnerabilities. You may also want to consult with the National Vulnerability Database to see if any of your packages have known security holes.
Steps
- Install cyclonedx/bom. The CycloneDX SBOM Specification is a format for listing packages in a node library and any other software project. cyclonedx/bom is a tool for generating a sbom that conforms to the CycloneDX SBOM Specification.
npm install -g cyclonedx/bom
- Use npx to run it. Navigate to the directory of your node project and run the following. This tells the package to generate a bill of materials by analyzing the node_modules folder in the current folder and save the output to bom.json.
npx @cyclonedx/bom . -o bom.json
- Install viewbom. Viewbom is a simple npx tool I wrote that generates an html UI for navigating the bill of materials you just generated.
npm install -g viewbom
- Run viewbom on the sbom you created.
npx viewbom bom.json bom.html
- Open bom.html in a web browser. This should present you with a simple UI that shows you some basic statistics about your sbom and give you a way to search the packages in it.
It should look something like the following screenshot.
How to set up stable diffusion on Windows
In this post, I'm going to show you how you can generate images using the stable diffusion model on a Windows computer.
How to set up stable diffusion on Windows
This tutorial is based on the stable diffusion tutorial from keras.io. I started with that tutorial because it is relatively system agnostic, and because it uses optimizations that will help on my low powered Windows machine. We only need to make a few modifications to that tutorial to get everything to work on Windows.
Steps
- Install python. First, check if python is installed using the following command:
python --version
If you already have python, then you can skip this step. Otherwise, type 'python' again to trigger the Windows Store installation process.
- Install pip. You can install pip by downloading the installation script at https://bootstrap.pypa.io/get-pip.py. Run it using the following command.
python get-pip.py
- Enable long file path support. This can be done in multiple ways. I think the easiest is to run PowerShell as administrator, then run the following command:
New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force
- Install dependencies. I had to manually install a few dependencies. Use the following command to install all needed dependencies.
pip install keras-cv tensorflow tensorflow_datasets matplotlib
- Save the code to a file. Save the following code into a file called whatever you want. I called mine 'stable-diffusion.py'.
import time
import keras_cv
from tensorflow import keras
import matplotlib.pyplot as plt
model = keras_cv.models.StableDiffusion(img_width=512, img_height=512)
images = model.text_to_image("california impressionist landscape showing distant mountains", batch_size=1)
def plot_images(images):
plt.figure(figsize=(20, 20))
for i in range(len(images)):
ax = plt.subplot(1, len(images), i + 1)
plt.imshow(images[i])
plt.axis("off")
plot_images(images)
plt.show()
- Run the code.
python stable-diffusion.py
- Enjoy. To change the prompt, simply alter the string that is fed into 'model.text_to_image' on line 8. Here are my first images created using this method and the given prompt.
How does the internet work?
In this post, I'm going to give you a simple explanation of how the internet works. This is the first in a series of posts that introduce readers to modern web programming.
TLDR: It's just computers talking to each other
When using the internet, one device requests a piece of information that is stored on a computer connected to the internet, and that computer returns it to the requesting device. The modern web is much more complex than that, but at it's simplest level it's just computer A requesting information from computer B.
The requesting device is usually called the client. The computer that serves the request is usually called a server. So if you have to remember one thing about how the web works, just remember this:
Clients request information that is stored on servers. Servers return that information to the clients.
What is a client?
The "client" we're actually talking about is a web browser. The web browser knows how to do several things. Here's some of the things that a web browser can do.
- Build a request for information using a language, or protocol, called HTTP.
- Send that request to servers on the internet by using the operating system to access network resources.
- Parse a response to an HTTP request that sometimes includes a web page.
- Display a web page made of html, css, and javascript.
- Interpret events fired by the operating system that come from devices such as keyboards, touchscreens, and mice.
That's just a bare fraction of what the client does, but it shows that modern web browsers are technical marvels.
What is a server?
The job of the server is slightly different. The server runs a piece of software that listens snd responds to incoming web requests. Such software is called a web server and it includes Apache, Node, and nginx. When it receives a web request, it seeks the resource that corresponds to the requested information to try to fulfill the request.
In the very early days of the web, the resources were just files. So a client might make a request for "/schedule.txt" and the server would see if it had such a file, and return it if it did.
How does the client know where to find the server?
Before the client can make a request to a server, it has to know which server has the information that it is seeking. It does this by using a service called Domain Name System, or just DNS for short.
Before the web browser makes the main web request, it makes an initial request to your internet service provider, or ISP, to ask where to find that information. So let's look at how this might work when you request "google.com".
- The web browser transforms "google.com" into "http://www.google.com". The "http" stands for Hyper Text Transfer Protocol, and that's the format of the request. The "www" is just an old fashioned naming system that stands for "World Wide Web". When someone sets up a "www" subdomain, they're essentially saying that that address should be publicly available on the internet.
- The web browser then sends a request to your DNS to learn the address for "http://www.google.com". That request goes to a DNS server, which responds with an IP address, which is essentially like a telephone number for a computer. So the request for the address for "http://www.google.com" might return something like "123.456.78.910".
- The web browser then sends a request to the indicated IP address, "123.456.78.910". Then the server does whatever it needs to do to fulfill the request, more on this later, and returns the requested information to the client.
- The client renders the page returned from the server.
Okay, so this is getting a little bit more complex, but it still follows the same model. The web browser first sends a web request to the DNS, then it sends the web request for the specific piece of information.
What does the server actually do?
There is a lot that I am glossing over in this explanation, because I'm trying to keep it simple. I will write future posts that go into a little more detail about how the client and server work, and show how code comes into play in those processes.
But we can break the action of the web server into several steps.
- It validates the request to ensure that it's a request that it can fulfill.
- It locates the requested resource. If no resource is requested, then it seeks a default, which is usually "index.html".
- It reads the requested resource, checking to see if it needs any additional information.
- It uses attached data sources such as databases or caches to request information needed to render the resource.
- It builds the final response incorporating the requested resource as well as any data requested from data sources.
But how does it actually put all of this together? It's the job of a web programmer to tell the server how to assemble a web page that fulfills the user's request.
I want to restate that because that's why we study the web. We care about how the web works so that we can contribute to making it more useful.
It's the job of web programmers to write instructions for web servers to fulfill web requests.
Summary
In this post, I gave a concise overview of how the internet works. Here are the main takeaways.
- When using the internet, a client makes a request for a resource that is stored on a server.
- The client knows how to make web requests and render the response using html, css, and javascript.
- The server uses web software to build web pages using by filling in information from connected data sources.
- DNS servers tell web browsers where to find the web address they are seeking.
More
How to set up a computer for web development
In this post, I'm going to suggest a basic set of tools required for web development. These instructions are aimed at people who are using a desktop computer for web development.
TLDR: Install Google Chrome, Visual Studio Code, and npm
Here are links to the three tools that I think are vital to web development.
What tools are needed for web development?
A programmer of any sort needs two basic tools:
- A tool to write code.
- A to to run the code they have written.
Code is usually represented as text, so to write code you will need a text editor. Today most programmers use a text editor that has been adapted for the task of writing code called an Integrated Development Environment, or IDE. The IDE that most programmers prefer in 2023 is Visual Studio Code. It's developed by Microsoft, who has a long history of creating high quality products for developers.
Traditionally, code is compiled and run by a computer's operating system, however, many modern systems take code written by a developer, then run it live using a tool called an interpreter. In this series, we're going to be writing JavaScript code, then running it in the interpreter in Google Chrome or in Node.
So those are the three tools we need to install to setup a basic environment for writing code for the web:
Node also includes a tool that is fundamental to web development called Node Package Manager, or NPM for short. NPM is a tool for downloading chunks of code written by other programmers, or packages, and using them in your own app.
More
Here are some other resources and perspectives on setting up your computer for web programming.
Introduction to HTML
In this post, I'm going to introduce HTML, and demonstrate how it acts as the skeleton for all pages on the internet.
TLDR: HTML defines the structure of the page
HTML is used to define the structure of a web page. It's the structure that your styles (css), and code (JavaScript) sits on top of. The CSS styles the html elements. The code allows interaction with html elements.
What is HTML?
HTML stands for Hyper Text Markup Language. The unusual phrase "hyper text" means "more than text". Hyper text can be contrasted to plain text. You read plain text in a printed book. The words string together to form a story. But what if you want to know more about a topic? What if you want a definition for a term? What if you want to email the author? HTML gives web programmers the ability to "mark up" the text in a way that allows web browsers to add additional functionality to a text document.
HTML is one of the key inventions that enabled the web to become so popular. The ability to add cross-references directly in a document enables all sorts of connections and interactions that were formerly impossible. Nearly every page on the internet is made using HTML.
What are HTML elements?
The classic example of a hyper text element is a link. When links were invented, they were called "anchors", as if the author is anchoring a bit of text to another document. When you want to add a link to an html document you use the anchor or "a" element. Here's an example.
<a href='https://www.myblog.com'>My blog</a>
Wow, that's pretty confusing, isn't it? But take a moment to look more closely. What parts of it do you see?
An html element can have an opening tag, any number of attributes, content, and a closing tag, as shown in this diagram.
Here's another example of an html element. It's the image, or "i" element.
<img src="https://www.evanxmerz.com/images/AnchorTag_01.jpg" alt="Anchor tag diagram"/>
Notice that the image element shown here doesn't have a closing tag. This is the shortened form of an html element that doesn't need any content. It ends with "/>" and omits the content and closing tag.
The structure of an html document
HTML documents are composed of numerous html elements in a text file. The HTML elements are used to lay out the page in a way that the web browser can understand.
<!DOCTYPE html>
This must be the first line in every modern html document. It tells web browsers that the document is an html document.
<HTML>
The html element tells the browser where the HTML begins and ends.
<HEAD>
The head element tells the browser about what other resources are needed by this document. This includes styles, javascript files, fonts, and more.
<BODY>
The body element contains the content that will actually be shown to a user. This is where you put the stuff that you want to show to the user.
An example html document
Here's an example html document that contains all of the basic elements that are required to lay out a simple HTML document.
I want you to do something strange with this example. I want you to duplicate it by creating an html document on your computer and typing in the text in this example. Don't copy-paste it. When you copy-paste text, you are only learning how to copy-paste. That won't help you learn to be a programmer. Type this example into a new file. This helps you remember the code and gets you accustomed to actually being a programmer.
What about typos though? Wouldn't it be easier and faster to copy-paste? Yes, it would be easier and faster to copy-paste, however, typos are a natural part of the programming experience. Learning the patience required to hunt down typos is part of the journey.
Here are step-by-step instructions for how to do this exercise on a Windows computer.
- Open Windows Explorer
- Navigate to a folder for your html exercises
- Right click within the folder.
- Hover on "New"
- Click "Text Document"
- Rename the file to "hello_world.html"
- Right click on the new file.
- Hover over "Open With"
- Click Visual Studio Code. You must have installed Visual Studio Code to do this.
Now type the following text into the document.
<!DOCTYPE html>
<html>
<head>
<title>Hello, World in HTML</title>
</head>
<body>
Hello, World!
</body>
</html>
Congratulations, you just wrote your first web page!
Save it and double click it in Windows Explorer. It should open in Google Chrome, and look something like this.
There are many HTML elements
There are all sorts of different html elements. There's the "p" element for holding a paragraph of text, the "br" element for making line breaks, and the "style" element for incorporating inline css styles. Later in this text we will study some additional elements. For making basic web pages, we're going to focus on two elements: "div" and "span".
If you want to dig deeper into some other useful HTML elements, see my post on other useful html elements.
What is a div? What is a span?
That first example is pretty artificial. The body of the page only contains the text "Hello, World!". In a typical web page, there would be additional html elements in the body that structure the content of the page. This is where the div and span elements come in handy!
In the "div" element, the "div" stands for Content Division. Each div element should contain something on the page that should be separated from other things on the page. So if you had a menu on the page, you might put that in a div, then you might put the blog post content into a separate div.
The key thing to remember about the "div" element is that it is displayed in block style by default. This means that each div element creates a separate block of content from the others, typically by using a line break.
<div>This is some page content.</div>
<div>This is a menu or something.</div>
The "span" element differs from the div element in that it doesn't create a new section. The span element can be used to wrap elements that should be next to each other.
<span>This element</span> should be on the same line as <span>this element</span>.
An example will make this more clear.
Another example html document
Again, I want you to type this into a new html file on your computer. Call the file "divs_and_spans.html". Copy-pasting it won't help you learn.
<!DOCTYPE html>
<html>
<head>
<title>Divs and spans</title>
</head>
<body>
<div>
This is a div and <span>this is a span</span>.
</div>
<div>
This is another div and <span>this is another span</span>.
</div>
</body>
</html>
Here's what your page should look like.
It's not very exciting, is it? But most modern webpages are built using primarily the elements I've introduced to you on this page. The power of html isn't so much in the expressivity of the elements, but in how they can be combined in unique, interesting, and hierarchical ways to create new structures.
Summary
In this section I introduced Hyper Text Markup Language, aka HTML. Here are the key points to remember.
- HTML is the skeleton of a web page.
- HTML elements usually have opening and closing tags, as well as attributes.
- HTML documents use the head and body elements to structure the page.
- Div and span elements can be used to structure the body of your webpage.
More
Introduction to CSS
In this post, I'm going to introduce CSS, and demonstrate how it can be used to apply styles to a web page.
TLDR: CSS defines the look and feel of a web page
After building the structure of a page using HTML, you can add styles to HTML elements using CSS. If HTML is a car's frame, then CSS is the body shape, the paint job, and the rims.
What is CSS?
CSS stands for Cascading Style Sheets. Programmers and web designers use CSS to describe the colors, fonts, and positions of HTML elements. Web browsers interpret the CSS on a page to show those styles to viewers.
CSS can be added to an HTML document in multiple ways. It can be added directly to HTML elements using inline styles, it can be added to the head section using style elements, or it can be included from another file.
A simple inline css example
Here's an example of a very simple use of inline css using the style attribute on an element. All it does is set the font color to blue.
<div style="color: blue;">This text should be blue.</div>
To try this in your own html file, just type that code within the body section of the page. If you did it right, then the text should appear blue.
There are three important parts of this inline css. The first is the style attribute on the div element. Use the style attribute to apply inline styles. The second is the word "color", which specifies which property of the element that should be styled. The third important bit is the word "blue", which specifies the color. There are many ways to specify colors in CSS, and using names is the simplest.
I'm introducing inline styles here because they are useful for debugging, however, inline styles are impractical for a number of reasons, so we're going to look at a few other ways of incorporating CSS into an HTML document.
A simple CSS style element example
CSS styles can also be included in a document using the HTML style element. Usually such styles are placed in the head section of the document.
Using the HTML style element like this is generally discouraged, however, because the styles are not reusable in other documents. That's why styles are usually stored in separate files that can be included in any HTML document.
It's good to know about style elements, however, because they are sometimes used in emails. When creating a beautiful email, you don't have access to other files, so style elements are sometimes the best option.
<style>
span {
background-color: red;
}
</style>
This CSS rule sets the background color to red for all span elements on a page. Here's how it might look in a full HTML document.
<!DOCTYPE html>
<html>
<head>
<title>Divs and spans</title>
<style>
span {
background-color: red;
}
</style>
</head>
<body>
<div>
This is a div and <span>this is a span</span>.
</div>
<div>
Notice that <span>all span elements on the page receive the style indicated in the rule above</span>. This is due to how CSS selectors work.
</div>
</body>
</html>
Here's a graphic showing how that style breaks down into parts. The selector is used to select which elements will recieve the style. The property and value describe the style to apply, just like when using inline styles.
What are selectors?
The problem with inline CSS is that it clutters up the code for your webpage, and it's not reusable. A single style declaration can be twenty or more lines long. That's a lot to insert into an HTML element, and adding it makes the code difficult to read. Also, inline styles only apply to the element they are on. If you want to use an inline style twice, then you have to duplicate it, which is a big no-no in programming. Fortunately, there are other ways to write CSS.
CSS can be included in a "style" HTML element. CSS written in the style element uses syntax called "selectors" to apply styles to specific elements on a page.
For example, say you wanted all span elements on your page to have a larger font. You might include something like the following in the head section of your HTML document.
<style>
span {
font-size: 2em;
}
</style>
That selector selects all elements with a given tag name. You could use "div" to style all divs, or "h1" to style all h1 elements. But what if you want to be more specific what if you only want to style some elements of a given type?
If you want to be more specific, then you can add class or id attributes to your html, then use those attributes in your CSS selectors.
Here's an example that shows a class attribute on a div and a selector that uses the class value. Notice that the class name is preceded by a period in the selector.
<style>
div.blueText {
color: blue;
}
</style>
<div class="blueText">This text should be blue</div>
Here's an example that shows how to use the id attribute and selector. Notice that the hashtag symbol precedes the id in the css selector
<style>
#blueText {
color: blue;
}
</style>
<div id="blueText">This text should be blue</div>
Usually class is used when you want to have many things that share the same style. The id attribute is used when you are styling something that is unique on the page.
Including CSS in a separate file
Inline styles and style elements are not the most common way of including styles in a web page. Usually styles are imported from a stylesheet file that contains CSS and ends with the .css extension.
Type the following code into a file named css_example.css.
span.blue {
color: blue;
}
span.red {
color: red;
}
#title {
font-size: 2rem;
}
Now type the following code into a file named css_example.html.
<!DOCTYPE html>
<html>
<head>
<title>CSS Example</title>
<link rel="stylesheet" href="css_example.css">
</head>
<body>
<h1 id="title">This is a large title</h1>
<div>
This is a div with <span class="red">red text</span>.
</div>
<div>
This is another div with <span class="blue">blue text</span>.
</div>
</body>
</html>
If you did it all correctly, then you should end up with something like this, when opened in a web browser.
What are all the CSS properties?
CSS can be used to style just about any aspect of an HTML element. With CSS you can create shadows, animations, and nearly anything else you see on the web. In my posts I'm going to focus on the most common CSS properties, but if you want to see the complete list of things that can be styled with CSS, then check out MDN's comprehensive list of CSS properties.
Summary
- CSS is the style layer of the web.
- CSS is usually imported into an HTML file from a separate CSS file.
- The element, class, and id selectors allow you to target your styles to specific elements.
More
CSS is quite large and complex, but you don't have to know everything about it to use it effectively. You only need to know the fundamentals, and be able to use web search to find the other stuff.
Project: Personal web page
This post contains the first assignment in my free web programming course.
Project: Personal web page
Now that you know a little html and css, you should be able to produce some simple web pages. In this assignment, I want you to create a web page about yourself. It should be pretty similar in content to my about-me page on my website, but it should be about you and the things you like.
Here's what it must contain.
- A title that says your name and is styled like a title.
- A picture of you or something you like.
- A short paragraph of at least three sentences. You can use Lorem Ipsum if you can't write three sentences about yourself.
- A quote you like that is styled like a quotation. Usually quotes are indented more and displayed in italics.
- At least three links to things you've done or things you like. Use html lists to show these.
Other useful HTML elements
In this post, I'm going to introduce some HTML elements that are still useful in the modern web development, even if they aren't used as often as div or span.
TLDR: These HTML elements are useful for accessibility and search engine optimization
You can build most webpages using only div and span elements, along with the structural elements html, head, and body. However, it's important to know about other elements if you want your page to rank highly in Google search results, or you want your page to be accessible to people with disabilities.
Headings and titles
Heading elements are used to show titles or subtitles. They are important to your visitors because they make text more readable. They are important to Google because Google uses them to understand the content of your pages.
Heading elements are h1, h2, h3, h4, and h5. The h1 element is the top level title of the page, and should only occur once on a page, otherwise Google Search Console will show a warning for that page, and you won't rank as highly in web searches.
Here's an example of an h1 element.
<h1>I'm a page title that can be seen by the visitor</h1>
It's important to note that there IS an actual title element in HTML. When the title element appears in the header, it is used to set the text that is shown in the tab of your web browser.
<title>This is the text at the top of the tab</title>
Paragraphs in HTML
Before CSS was invented and standardized, there were lots of HTML elements used to format text. You can still use some of these, but it is widely discouraged. The "b" element bolds the enclosed text. The "i" element applies italics to the enclosed text. The "br" element adds a newline. You should never use any of those elements, but there is one element that is still quite useful.
The "p" tag is used to enclose a paragraph of text. It is useful as something separate from a div because it includes useful browser default styles, and because it serves as a good way to style the body text for your page.
<p>I'm a paragraph with some nice browser default styles.</p>
Browser defaults are a complex and controversial topic. Each browser starts with its own set of default styles so that a basic HTML page will be readable. There are many modern packages that override or eliminate these styles so that they don't interfere with a site's custom styles. I don't have a strong opinion on browser defaults in either direction.
Lists in HTML
Lists are another bit of semantic HTML that are still widely used. There are two types of lists in HTML, ordered lists and unordered lists. These can be represented by the "ol" and "ul" elements respectively. List items can be represented by the "li" element.
So what's the difference between the two list types? Ordered lists are numbered, while unordered lists are bulleted.
<ol>
<li>First item</li>
<li>Second item</li>
</ol>
If you drop that into an HTML file then you will see something like the following.
I encourage you to replace the "ol" elements with "ul" and see what happens.
Nav, main, and footer elements in HTML
Finally, I want to recommend using the elements nav, main, and footer. These elements are used to encapsulate the navigation portion of the page, the main content of the page, and the footer of the page. These elements are absolutely VITAL for accessibility, as they tell screen readers about which part of the page they are reading. You need to remember accessibility for two reasons. It is important to allow as many people as possible to access your site, and Google will penalize you if you don't.
They are used just like divs.
<nav>
<ul>
<li>Menu item 1</li>
<li>Menu item 2</li>
<li>etc</li>
</ul>
</nav>
Introduction to JavaScript
In this post, I'm going to introduce the JavaScript programming language and show you a few ways it's commonly used on the web.
WARNING: Some prior programming required!
I'm not introducing programming in general. This isn't a good first tutorial for someone who has never written a line of code in their life. I'm assuming that you know what functions are, and that methods and subroutines are the same thing. I'm assuming that you've written at least a little of some kind of code in the past.
TLDR: JavaScript is the engine of a webpage
HTML is used to define the structure of a web page. CSS defines the look and feel of a page. JavaScript defines how a page acts and responds to user input.
If a web page is a car, then HTML is the frame, CSS is the body, and JavaScript is the engine.
A haiku about JavaScript
Dearest Javascript,
You are my favorite tool
that costs me nothing.
Why not Java?
JavaScript has nothing to do with Java. The name JavaScript came about as an attempt to steal some of the hype around the Java programming language in the 1990s. In the 30 years since then, their positions have entirely flipped, with JavaScript now being the programming language that sees tons of innovation and market support, while Java is withering. Why has this happened? Why has JavaScript been so successful?
In my opinion, Java has been uniquely unsuccessful as a language because it has refused to grow and innovate. Java set some standards and some conventions for the language, and even thirty years later, the committees who guide specification for that language continue to be far too conservative.
If you're a Java programmer and you don't agree with me, then I encourage you to try Microsoft C#. That's the fun language that Java could have been.
And why is that?
C# and JavaScript have become incredibly open-ended. You can write code in lots of different ways, using features borrowed from other cool languages and frameworks. Both are sort of like Frankenstein's monster inasmuch as they are collections of mismatched parts that come together to form a surprisingly intimidating beast.
And this scares some people.
But, much like Shelley's actual monster, they are actually quite nice, and I hope to convey that to you in this introduction.
But what is JavaScript?
It's a programming language that runs in the web browser and on nearly every device that exists. On a webpage, you include JavaScript within the script element, or by including code in an external file. There's more to it than that, but let's learn by doing.
A JavaScript function
One of the key responsibilities of JavaScript is to respond to user input. Beside a few basic interactions, like clicking a link or a dropdown, every action that occurs on a webpage is handled by custom event handlers that are usually written in JavaScript. In this case, we're going to write a script that handles a click event.
When the user clicks on something, the browser creates what is called a click event. It then runs a bit of code called a function that is tied to that event. Here's the click handler function that we're going to write.
function showWarningOnClick() {
alert("Warning! The elders of the internet have been notified!");
}
That might look like nonsense to you, so let's break it down. This graphic shows how each of the pieces come together.
The first line is the function declaration. The word "function" declares that the next word is the name of a function. So "showWarningOnClick" is the name of the function. The parentheses, "()", would enclose the method arguments, if we needed any. The open bracket, "{", indicates that the next line will be the first line of the function body.
The second line is the actual code that is executed by this function. In this case, it calls another function. It calls "alert", with the argument "Warning! The elders of the internet have been notified!". Notice that the argument is enclosed with parentheses. Then the line ends with a semicolon. Most lines of JavaScript code must end in a semicolon.
That's the code that we will run when a button is clicked. We will see in the next section how we can trigger that function using a button.
A JavaScript event
The following line of code creates a button and adds a handler for the onClick event.
<button onClick="showWarningOnClick();">Don't click me</button>
Notice that the button isn't created using a div or span element, as you might expect. When creating buttons, it's good to use the button element if you can. This element is friendly to screen readers, and other accessibility technology.
The only attribute given is "onClick", and it's set to call the function that we defined earlier. The onClick attribute tells the browser what to do when the button is clicked.
It's important to note that onClick events can be attached to any elements. They aren't something unique to buttons. You can attach methods to the onClick attributes of divs or spans, or any other HTML elements.
Responding to a click event in JavaScript
So let's put it all together. The code should look something like this. And as a reminder, please type this in, don't copy paste it. The point is to learn, and you will learn and remember more if you type it out.
<!DOCTYPE html>
<html>
<head>
<title>JavaScript Hello World</title>
<script>
function showWarningOnClick() {
alert("Warning! The elders of the internet have been notified!");
}
</script>
</head>
<body>
<div>
<button onClick="showWarningOnClick();">Don't click me</button>
</div>
</body>
</html>
When you run it, by double clicking it and opening it in a web browser, you should see a single button in the upper left that says "DOn't click me". Click it, and you should see a little popup with the message "Warning! The elders of the internet have been notified!".
Summary
JavaScipt is a fun, open-ended programming language that is particularly useful in web browsers. In this post we learned about JavaScript functions and event handlers.
More
Here are some other introductions to JavaScript from around the web. I suggest that you spend time to check them out, even if you found this really easy. It's good to go over the fundamentals often, and it's particularly good to get multiple perspectives.
- MDN introduction to JavaScript. I link to MDN in every post because it's the most reliable source.
- Introduction to JavaScript from javascript.info.
- Introduction to JavaScript from freecodecamp. Some of what he says in this video is a bit misleading, but it's still a good place to start.
How to Debug a webpage in Chrome
In this post, I'm going to show you several ways to debug javascript in Google Chrome.
TLDR: Shift + ctrl + c
Open the javascript console using the shortcut shift + ctrl + c. That means, press all three at the same time. If you're on a Mac, use cmd rather than ctrl.
Inspecting an element to add new styles
Open the dom example that we wrote in What is the DOM?. In this post, we'll use that page to explore three ways to find out what's going on in a webpage. The code should look something like this.
<!DOCTYPE html>
<html>
<head>
<title>How to modify the DOM using JavaScript</title>
</head>
<body>
<div id="post">
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</div>
<script>
let blogPost = document.getElementById("post");
let copyrightNotice = document.createElement("p")
copyrightNotice.innerHTML = "Copyright " + new Date().getFullYear() + " by the author";
blogPost.append(copyrightNotice);
</script>
</body>
</html>
After opening it in Chrome, right click on the copyright notice, then click "Inspect" in the popup menu. The html for the element you clicked should appear in the "Elements" tab of the browser console. It should look something like this but there are several places where the console can appear.
Any time you are confused about an element, inspecting it is a good place to start. In the browser console, you should be able to see the html for the element, as well as any attached css. In fact, you can actually add new styles using the browser console. Notice in the "Styles" tab where it says "element.style" and there are empty braces. Any CSS you type in there will be applied to the selected element. Styles can also be toggled on and off. This can be exceptionally handy when trying to figure out why your page doesn't look quite right.
Go ahead and try it. Enter "font-weight: 800" into element.style. What happens?
Debugging using console.log
The most basic way to debug JavaScript is to put some console.log statements into your code, then open the console and see what they print. Calling console.log tells the browser to print whatever you pass in in the JavaScript console. You can console.log just about anything, but the web browser will format variables automatically if you put them in an object using curly braces.
So the following console.log call isn't very useful.
console.log('HERE!');
Whereas the following console.log call will allow you to examine the object passed int.
console.log({ confusingVariable });
Add the following line of code to the dom example after it sets the copyrightNotice variable.
console.log({ copyrightNotice });
Then reload the page and use shift + ctrl + c to open the browser console, or shift + cmd + c on a Mac. You may have to expand the window and click over to the tab that says "Console".
You should see something that says "{ copyrightNotice: p }", and if you expand it, then you should be able to see all the properties associated with that variable. It's not particularly useful in this simple example, but I use this technique virtually every day as a web programmer.
Debugging using the Chrome debugger
Keep the dom example open in Chrome. Then press shift + ctrl + c to open the browser console. Navigate over to the "Sources" tab. This tab is the built in debugger provided by Google Chrome. In this debugger, you can add breakpoints to your code, and the browser will stop at the breakpoints and allow you to examine the state of your code. It's okay if this looks very confusing. We don't need to do much with it.
There are three things you need to be able to do. First, you need to be able to navigate to the sources tab. Then you need to find the line numbers, because you need to click them. This is much more difficult in a typical web page that has many files. Finally, you need to know how to continue running the page after it stops at your breakpoints.
After you've opened up the page, click line 12 to set a breakpoint there. In this case, that code has already been run, so reload the page to make it run again. When you do so, you should see something like the following.
Notice the area that I've circled in that screenshot. It's where you can see the state of your code. All of your variables should appear there and you should also see their values. Notice that copyrightNotice is undefined in this case. That's because Chrome pauses before running the code at the line where you placed your breakpoint. If you place a breakpoint on the next line, then press the play/pause button to resume execution, the value for copyrightNotice should be filled in.
The Chrome debugger is a great way to examine the state of your code.
Project: Client-side web form
This post contains the second assignment in my free web programming course.
Project: Client-side web form
Forms are how users enter data on the web. When you register for a website, you enter your username and email into a web form. When you buy an item from a web store, you send your payment information using a web form.
Generally speaking web forms have input components on them like text boxes and dropdowns and buttons. But this isn't always the case. Web forms can be cleverly hidden using JavaScript and CSS.
Typically a web form submits data to a server, which processes that data and responds to the user with something interesting. For instance, you might submit your username and password and get back an authentication token that allows you to use the private portion of a website.
Requirements
I want you to build a single web page using html and javascript only.
It should have three things on it.
- A page title in h1 tags.
- A label that reads "title:".
- An input text box.
- A submit button.
When the button is clicked, the title of the page should change to the title entered by the user without reloading the page.
Also, the user input must validated. The user should be prevented from using dirty words like "MacOS" and "iphone". Don't let them use either of those words in any capitalization. If those words occur in the input, then a red error message should be displayed underneath the text box that reads "Please refrain from using dirty words such as 'MacOS' and 'iphone'.".
Extra credit
For extra credit, make the form beautiful using CSS. Be creative, but try to make something that a typical web user might like.
Goodbye Gatsby! Hello Eleventy!
For the last few years, I had been running this blog using Gatsby. In general, I liked Gatsby. I liked that it was so simple up front, but had such deep complexity under the hood. I liked that it supported React out of the box. I liked that it could generate websites in so many different ways.
I liked it enough that I made my own minimalist blog starter called Empress, which was a fully featured blog based on the tutorial but with several key features added. I used that for this blog for several years and I liked it.
But man, the leadership of Gatsby was bad. They took it in such weird directions. They tried and failed at so many weird features that the documentation, and the product itself just became a dumping ground.
It really went off the rails when they created Gatsby "themes" that had nothing to do with the colloquial use of "themes". Themes were really more like plugins, but they were already using that term, too. So it was a hot mess, and soon enough the feature seemed to disappear from the docs and from existence.
Then they were bought by Next. I still don't understand that acquisition. Next.js is a good product with sensible leadership. I liked using it at my prior job, and I considered using it for this site, but it was just overkill.
So when I could no longer navigate the crazy web of dependencies to upgrade my blog to the latest version of Gatsby, I essentially stopped writing. I was just so frustrated with it.
Until this weekend, when I decided it was time to change.
Hello, Eleventy!
I decided to switch to Eleventy mostly because it's the static site generator with the most hype right now. Sometimes it's good to try the tech that is currently being hyped. It's good to stay up to date and have that line on your resume.
But also, Eleventy is a good product. I planned on spending a few weekends getting my blog switched over, but it only took me a few hours on Saturday, Sunday, and Monday to get it 95% switched over. There are still a few minor bugs here and there - don't check the title tags on tag pages, and ignore the reverse chronological sorting in some places - but in general the switch was very easy.
I've been around web programming since the 1990s, and Eleventy feels like a return to an older way of working, but with some of the good infrastructural tech we picked up along the way. Like now I'm writing in markdown, and I don't need to run MySQL on my server, nor do I even need to run a traditional "server". I can just host my static site on S3, and use GitHub Actions to deploy it, just like with Gatsby.
And I've heard some of the blowback against Jamstack architecture sites in the last few years. It's why I briefly considered using Next for this. But I still think JavaScript + API is the best/fastest/cleanest way to run a website, whether it's a blog or something more complex. I don't have an API for this site, but I feel like I could attach one without too much trouble if the need arose.
I do wish that I could use React. I saw some people try to get Eleventy working with React experimentally, but I didn't see any updated package to do it. Nunjuks and markdown are okay for the basics, but I do think that a true component-based architecture is ultimately a cleaner way to build UIs than a template based architecture.
Until next time...
So I'm going to give Eleventy a shot for now.
But we all know that those of us who still quixotically maintain personal blogs in 2024 are on a perpetual upgrade cycle. We all know that no "upgrade" is the final upgrade.
I wonder what I'll "upgrade" to in a few years? Maybe Wordpress will come back around?