Despite the fact that I’ve been working full time in software for almost 8 years now every now and then I still need a reminder of how useful reading logs can be in helping solve problems.

I had a couple of such instances recently which I thought I’d document.

The first was a couple of weeks ago when Tim and I were pairing on moving some applications from Passenger to Unicorn and were testing whether or not we’d done so successfully.

We were doing this by creating an /etc/hosts entry from our top level domain to an nginx proxy node which was to forward on the request to the application server.

This didn’t work and we got a 404 response code so I logged onto the server hosting the application server and started writing our a cURL command to simulate what the proxy should be doing to see if the problem was there.

After watching me do this a couple of times Tim suggested that we might be more successful if we opened a bunch of tabs on the shell tailing the various log files that the request should pass through.

We set up tail commands against the following files:

Immediately it became clear that we actually had a problem on the proxy node because we’d configured one of the nginx directives incorrectly.

Once we’d fixed this the request flowed through smoothly.

We extended this tailing of files idea when testing multiple nodes through a load balancer except this time we made use of Fabric to make things a bit easier.

The second was earlier this week when Baris and I were trying to configure puppet so that we could install different Ruby versions on different machines.

We were having trouble figuring out why the wrong version was getting installed so eventually we chatted to Andy who amongst other things had a look at the apt history log @ /var/log/apt/history.log and was able to figure out how this was happening.

Lesson of the last two weeks: embrace the logs!