My Little Weekend Project

Posted Sat Dec 10 @ 06:05:23 PM PDT 2011

Instead of studying for my finals, I played around with Google's Map API. I thought it would be cool to plot my website visitors on a map in near real time. I ended up using a combination of linux utilities, Perl, C, PHP, JavaScript and MySQL to achieve the desired effect.

Getting IP and Host from Apache

The first thing I needed to do was extract the visitor's IP and host (i.e. which website they were viewing on my server that hosts multiple websites). Apache's log file is the natural place to look for that information. Unfortunately, by default, Apache does not log the host. But it is easy to fix. All I needed to do was add %{Host}i to the LogFormat section:

If you follow the Apache log, you will see everything you need:

The reason I use --follow=name is because Apache swaps out the log file every once in a while, and I want to follow that filename, not the inode currently associated with the filename.

Sending Data Over Network

Here's the cool part. There is a linux command that makes it really easy to pipe stuff over a TCP connection (or UDP, if you like to live on the edge). It's called nc. Want to send that log file over the network? Piece of cake:

As long as you have nc listening on port 5000 on the other machine:

the log file will be piped out over the network onto the listening machine. Pretty cool.

Extracting IP and Host From Log (incoming over the network)

On the machine receiving the log, send the data to Perl to parse out the interesting parts:

That $|=1 code is used to disable buffering STDOUT (I don't want Perl to buffer the output, I want to see it right away).

Getting Latitude and Longitude from IP

The generous folks at MaxMind offer a free binary file (and C bindings) that map an IP address to a lat and long. It installs without a hitch on linux. I wrote a quick program, based on one of their examples, that would get the coordinates for the IP I parsed out in Perl:

Saving IP, Host, Lat and Long

I'll admit, this isn't the best way to store this streaming data, but it was pretty simple. I created a new database called IpQueue, and created a single table:

As my IP, host, lat and long tuples come out of my C program, I spit out an insert query (with an update):

Then I pipe that query to MySQL:

The entire pipe looks like this (it's a mess):

And we're still not done...

Outputting Points in HTTP

I made a simple page in PHP that outputs the most recent points stored in the database:

Google Maps

And finally we get to the Google map code. I simply query the PHP page roughly every second, and plot new points, and remove the old ones:

The end result is like this (I wish I could make this into an animation, but I don't know how):

Now I can study for my finals...

<< Home