Running varnishstat remotely using an API written in Go

Varnish is one of my favorite pieces of technology. It’s a reverse caching proxy that makes slow websites go fast. It comes with some useful binaries to measure and monitor a running instance.

Varnishstat is one of those binaries: it lists usage statistics of your local Varnish instance. And one statistic I’m very interested in is the current memory usage.

One of the biggest limitation of these binaries is that you can’t call them remotely, unlike what varnishadm and varnish-cli offer.

That’s why I wrote a small service the retrieves the memory usage and returns it through a RESTful webservice.

Calling varnishstat

Just run the varnishstat binary from the command line and it will start reporting on the status of the Varnish instance that runs on this machine.

You can also pass certain parameters that will allow you to filter and change the output format. The following command will run varnishstat, but will only display the following fields:

The number of cache hits
The number of cache bypasses
The number of cache misses
The number of objects in cache
The size of the cache in bytes

varnishstat -f MAIN.cache_hit -f MAIN.cache_hitpass -f MAIN.cache_miss -f MAIN.n_object -f SMA.s0.c_bytes

When you run the varnishstat binary, you’ll a continuously updated list of counters. That nice from a human readable point of view, but from a code/automation point of view, this is not really working for us.

The command below outputs the value only once and doesn’t return a continuously updated list of values:

varnishstat -f SMA.s0.c_bytes -1

So the “-1” parameter will make sure the output will only be listed once. The “-f SMA.S0.c_bytes” parameter will only list the amount of used bytes.

This is the output:

SMA.s0.c_bytes        66144000     53471.30 Bytes allocated

This is a step in the right direction, but JSON would be an ideal output format. Varnishstat supports JSON output and it’s just a matter of adding a “-j” parameter:

varnishstat -f SMA.s0.c_bytes -j

This is the output:

{
 "timestamp": "2015-08-27T15:25:51",
 "SMA.s0.c_bytes": {"type": "SMA", "ident": "s0", "value": 66144000, "flag": "a", "description": "Bytes allocated"}
}

The video below gives you a more accurate and visual representation:

I wrote it in Go

You might have noticed that I mentioned the term “Go”. Yes, I did write it in Go. Why? Because Go is cool, Go is powerful, Go is a language on the rise. Go is a modern compiled language that was originally invented at Google. It’s open source, it’s statically typed, it’s very fast (thanks to built-in concurrency mechanisms) and surprisingly easy to use. Go is a true systems language.

GoJi

Have I mentioned I’m using GoJi. Probably not. GoJi is a web micro-framework written in Go and I’m using it to quickly write fast and effective RESTful APIs. You can compare GoJi to micro-frameworks in other languages like:

Sinatra for Ruby
Flask for Python
Silex for PHP

The idea is that you register you routes and attach callbacks. These callbacks execute the logic associated with that route.

The code

Without any further ado, here’s the code:

Running the code

Running the code isn’t that hard and there are 2 ways to do it:

Run as script using the Go runtime
Compile and run as static binary

Before we run it, we need to make sure all external dependencies are present. To achieve that we run the following command:

go get

If you run the script using the go runtime, you need to make sure the Go runtime is installed on your production server. The upside is that you don’t need to compile your code for the specific architecture. The downside is that you need to install yet another piece of software on the production machine. This is how you do it:

go run varnishusage.go

The other option is to compile the code (including all of its dependencies) as a single static binary. That way, you don’t need to install any extra software on the server where the script will be running. But be sure to compile it for the right architecture. This could actually mean cross-compiling for a different architecture.

Compiling it for your current architecture can be done with the following command:

go install

You’ll find the compiled binary in the bin folder of your go root directory.

If the architecture or operating system on which you develop your code differs from the production environment, you’re going to have to cross-compile your code. If you’re using Go 1.5, it’ll be pretty easy:

env GOOS=linux GOARCH=amd64 go build varnishusage.go

The video below shows you how to do it in more detail:

And finally it’s a matter of performing an HTTP call on port 8000 of your Varnish machine to retrieve the usage

Tech