Running varnishstat remotely using an API written in Go
Varnish is one of my favorite pieces of technology. It’s a reverse caching proxy that makes slow websites go fast. It comes with some useful binaries to measure and monitor a running instance.
Varnishstat is one of those binaries: it lists usage statistics of your local Varnish instance. And one statistic I’m very interested in is the current memory usage.
One of the biggest limitation of these binaries is that you can’t call them remotely, unlike what varnishadm and varnish-cli offer.
That’s why I wrote a small service the retrieves the memory usage and returns it through a RESTful webservice.
Calling varnishstat
Just run the varnishstat binary from the command line and it will start reporting on the status of the Varnish instance that runs on this machine.
You can also pass certain parameters that will allow you to filter and change the output format. The following command will run varnishstat, but will only display the following fields:
- The number of cache hits
- The number of cache bypasses
- The number of cache misses
- The number of objects in cache
- The size of the cache in bytes
varnishstat -f MAIN.cache_hit -f MAIN.cache_hitpass -f MAIN.cache_miss -f MAIN.n_object -f SMA.s0.c_bytes
When you run the varnishstat binary, you’ll a continuously updated list of counters. That nice from a human readable point of view, but from a code/automation point of view, this is not really working for us.
The command below outputs the value only once and doesn’t return a continuously updated list of values:
varnishstat -f SMA.s0.c_bytes -1
So the “-1” parameter will make sure the output will only be listed once. The “-f SMA.S0.c_bytes” parameter will only list the amount of used bytes.
This is the output:
SMA.s0.c_bytes 66144000 53471.30 Bytes allocated
This is a step in the right direction, but JSON would be an ideal output format. Varnishstat supports JSON output and it’s just a matter of adding a “-j” parameter:
varnishstat -f SMA.s0.c_bytes -j
This is the output:
{ "timestamp": "2015-08-27T15:25:51", "SMA.s0.c_bytes": {"type": "SMA", "ident": "s0", "value": 66144000, "flag": "a", "description": "Bytes allocated"} }
The video below gives you a more accurate and visual representation:
I wrote it in Go
You might have noticed that I mentioned the term “Go”. Yes, I did write it in Go. Why? Because Go is cool, Go is powerful, Go is a language on the rise. Go is a modern compiled language that was originally invented at Google. It’s open source, it’s statically typed, it’s very fast (thanks to built-in concurrency mechanisms) and surprisingly easy to use. Go is a true systems language.
GoJi
Have I mentioned I’m using GoJi. Probably not. GoJi is a web micro-framework written in Go and I’m using it to quickly write fast and effective RESTful APIs. You can compare GoJi to micro-frameworks in other languages like:
The idea is that you register you routes and attach callbacks. These callbacks execute the logic associated with that route.
The code
Without any further ado, here’s the code:
Running the code
Running the code isn’t that hard and there are 2 ways to do it:
- Run as script using the Go runtime
- Compile and run as static binary
Before we run it, we need to make sure all external dependencies are present. To achieve that we run the following command:
go get
If you run the script using the go runtime, you need to make sure the Go runtime is installed on your production server. The upside is that you don’t need to compile your code for the specific architecture. The downside is that you need to install yet another piece of software on the production machine. This is how you do it:
go run varnishusage.go
The other option is to compile the code (including all of its dependencies) as a single static binary. That way, you don’t need to install any extra software on the server where the script will be running. But be sure to compile it for the right architecture. This could actually mean cross-compiling for a different architecture.
Compiling it for your current architecture can be done with the following command:
go install
You’ll find the compiled binary in the bin folder of your go root directory.
If the architecture or operating system on which you develop your code differs from the production environment, you’re going to have to cross-compile your code. If you’re using Go 1.5, it’ll be pretty easy:
env GOOS=linux GOARCH=amd64 go build varnishusage.go
The video below shows you how to do it in more detail:
And finally it’s a matter of performing an HTTP call on port 8000 of your Varnish machine to retrieve the usage