Evan X. Merz

You can keep paying exorbitant fees to cloud LLM providers. I can't stop you. But, if you have a reasonably good computer, it's quite possible to run the models locally without paying a dime.

Will the model you run locally be as good as the best models offered by expensive cloud services such as Claude Code? No. Will they be sufficient to your purposes? Perhaps.

They also have the additional benefit of total privacy. You can disconnect your machine from the internet and ensure that everything you say to your LLM stays local on your computer.

How to run a model on your local machine using ollama

There are several ways to run a model locally. You could just use the Ollama app and pick a model from there, however, the app appears to be non-functional for me. I select a model and nothing happens. I send a chat and nothing happens.

So for now, you might need to do a little more than just download the app.

Here's how to run a model on your local machine using the Ollama Docker image.

1. Install docker.

Docker is a system for distributing software in virtual bundles called containers. It allows users to create and distribute an "image" that is essentially a virtual computer in one file. So one user can put an operating system and a bunch of software into a Docker image, then users who want to run that software can do so by running that image.

The easiest way to use Docker on Windows is to install Docker Desktop.

2. Pull the Ollama docker image.

Next, you will need to open a new command line window (command line = terminal = console). I use Windows PowerShell, but good old fashioned cmd should work just as well. You may need to install the WSL (Windows Subsystem for Linux). I can't recall if I needed to install that for this or not.

You will also need to have Docker running, which you can do simply by starting the Docker Desktop app.

Use the following command to pull the Ollama Docker image.

docker pull ollama/ollama

3. Run the Ollama Docker image.

Then you need to run the Ollama Docker image. This image will allow you to download and run models. You can do this in Docker Desktop using the UI, but I usually just do it in the console.

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

After you run this command once, the same container can be restarted using Docker Desktop.

4. Download and run the model you want.

The you need to select the model you want to run and run it. The smallest model is tinyllama.

docker exec -it ollama ollama run tinyllama

But the results from that model are pretty bad. I prefer the 1 billion parameter model from Meta, llama3.2:1b.

docker exec -it ollama ollama run llama3.2:1b

That should leave you with a command prompt where you can pose questions to your LLM.