Logfiles Made Interesting with glTail

Fudgie writes "My boss claimed it was pretty much impossible to create an entertaining way to visualize server traffic and events in a short time frame, so of course I had to prove him wrong. A weekend of neglecting my family produced a small ruby program which connects to your servers via SSH, grabs and parses data from Apaches access log and Ruby on Rails production log, and displays your traffic and statistics in real-time using a simple OpenGL interface (tested under Linux and Mac OS/X). It's a bit hard to explain over text, so please have a look at fudgie.org for an example movie, and more information."

Visitorville

[Anonymous Coward]

The most entertaining way I ever saw to view logs was Visitorville [visitorville.com]-its kind of like SimCity meets web logging.

Re:Looks promising

[Fudgie]

Anything put into a logfile could be parsed and shown. I've tried with emails, shoutcast listeners and server logins, but they're not as interesting to show in the movie as I don't have the kind of traffic to make it useful.

Wow !

[cheros]

Obligatory jokes about 'taking the piss' aside, that is brilliant. It's the ultimate 'machine that does ping' (to name an old sketch) to keep management amused, but also provides real data. I bet that screen will go ballistic when you get Slashdotted (also a good way to visualise DDoS, maybe?).

I was about to say that it's a sort of etherape on steroids, but I've just realised your visualisation could benefit etherape instead (if you don't know etherape, look it up. No tools identifies a virus infection quicker).

Class, I'm impressed.

just a ploy to visualize the slashdot effect

[molo]

Notice in the movie that one of the sites being monitored is fudgie.org, which is what is linked to here. This looks like a ploy to visualize the slashdot effect. :) Wonder what that must look like. Might tax the renderer pretty hard. I guess that is one way to get load testing done!

-molo

Re:just a ploy to visualize the slashdot effect

[Fudgie]

http://www.fudgie.org/slashdotted.jpg [fudgie.org] for how that looks.

Re:Postfix?

[Fudgie]

Shouldn't be too hard. I'll cook one up this evening.

Re:Pretty, but?

[fmobus]

I believe this sort of tool is useful for realtime monitoring of net resources utilization. It can assist you giving graphic clues when something goes out of the usual parameters, like DDoS, slashdotments (sp?), router failure, etc. Depending on information being monitored and how it is displayed, it could also be used for long-term decision like buying more hardware or switching software because the current setup is not handling the load.

One nice, but more local example is the "duck" activity monitor (a windowmaker classic): a duck floats by a mass of water. If the water gets to high, it means the memory usage is high; if it has too much bubbles, processor is being hit. No percentages nor text, just a simple graphic.

A place I used to work is now trying to develop something like this: visualizations where you can tell trouble is brewing in a glance. This is useful for them because their services involved a lot of maintenance of third-party networks but having someone dedicated to nanny all systems is "dumb" and error-prone. Their solution consists of multiple screens around the office showing how the systems they are responsible for are behaving.

Re:just a ploy to visualize the slashdot effect

[nacturation]

Very nice. One suggestion: rather than have each side's dots fall off at the bottom of the opposite side, how about matching up serving requests with the originating referral so that the dots go to the corresponding spot on the right? Also, if you're not familiar with Flight Patterns [ucla.edu] it's along the same lines. Borrowing from that, it'd be quite interesting to show a 2D map arranged in a hub and spoke model with the center being the site(s) and the spokes representing the top 10 (or 20... configurable) referring sites with a special case for search engines.

Well, perhaps I'll have to learn Ruby and hack this myself. The script certainly looks clean enough.
 

Re:Wait, what... they're not interesting?

[Fudgie]

A lot of my time at work is spent looking at logfiles from webservers, applications servers, and databases looking for things about to break down, but after I introduced this I just need to glance at a screen to instantly see if some server has stopped answering, is taking too long to answer, or is generating way more exceptions than normal. I also add an event (the login text bouncing down the screen in the movie) on each money generating activity, which always amazes marketing people when they walk by.

Re:just a ploy to visualize the slashdot effect

[Fudgie]

Interesting idea. Shouldn't be too hard to try something like that, I already have some code in there doing something similar meant for incoming emails, uploads and other data going into the servers/sites. Try adding :type => 5 to the URL activities for an example. -- Erlend