Data Visualization using Perl/Tk

Idean writes "Generating a visual representation is often the best way to understand large data sets, but standard tools such as gnuplot often fall short. This article shows how to use Perl/Tk, the standard GUI toolkit for Perl, to quickly build custom plotting and graphing tools."

Perl or Line Noise

[Mr. Darl McBride]

Perl/Tk. Because Perl wasn't cryptic enough. :)

I know, mod me down! But hear me out first: We played a game of "Perl or Line Noise" where lines of hand-picked valid perl (admittedly some more cryptic choices) and 7-bit chopped output from /dev/random were presented on screen. Three users had to race to punch a button first if they thought they were looking at valid perl. If it was "line noise," they had to let it time out after three seconds, just like Jack Attacks, in You Don't Know Jack.

At the end, two

Re:Perl or Line Noise

[Phillup]

And how did your control group do?

What language was Perl compared to using this methodology?

Did the test subjects have experience with Perl? And, what of the control group?

Perl is different, no doubt about it. But, it doesn't *have* to look like line noise.

Re:Perl or Line Noise

[Mr. Darl McBride]

We didn't have a control group. We were drunk and coming off some good gaming. There is no science in FUN!

Re:Perl or Line Noise

[el-spectre]

If you're a good programmer, you can read the line noise.

At the same time, a good perl hack wouldn't write unreadable code. I often rewrite a complex operation, breaking it into 3 or 4 step, out of courtesy to the poor schmuck who has to maintain the code.

Totally on Topic!

[Froze]

This must be one the few slashdot stories that is completely on topic, and by that I mean "News for Nerds, Stuff that Matters".

This will be a nice new toy for me in my work as a research student, Thanks!

Use plotting software instead

[CompVisGuy]

I don't see the point in using Perl and Tk to write you own scientific plotting software. There is already plenty of software out there to do this, and it's better written and tested than anything one would produce in an afternoon with Perl and Tk.

The last thing you need when exploring large and complex data sets is to find that the code you are using to visualise the data is buggy! Don't roll your own!

I use Matlab for my work, and it has a fantastic range of scientific plotting features. It's not cheap, but there are some good free/OS packages too:

Look at:

gnuplot (http://www.gnuplot.info/)
KMatplot (http://kmatplot.sourceforge.net/)
GGobi (http://www.ggobi.org/)
Gri (http://gri.sourceforge.net/)

There are others, too.

I do not consider the data that was used in the example to be complex or large, and I don't see how Perl could help, as numerical processing is certainly not its strength!

Perl may be useful for massaging data into a form that is accepted by a scientific visualisation package, however.

Re:Use plotting software instead

[spencerogden]

"and I don't see how Perl could help, as numerical processing is certainly not its strength!"

Have you explored PDL (piddle)? I find it to be a very powerful way to manipulated datasets.

Re:Use plotting software instead

[acaird]

Another entry in the "good free software for plotting that is like Matlab" catagory (yes, very specific), is Octave [octave.org].

From their web site: "GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language."

See also: PDL

[bill_mcgonigle]

I'll just copy what their homepage [perl.org] says:

PDL ("Perl Data Language") gives standard Perl the ability to compactly store and speedily manipulate the large N-dimensional data arrays which are the bread and butter of scientific computing.

PDL turns perl in to a free, array-oriented, numerical language similar to such commerical packages as IDL and MatLab. One can write simple perl expressions to manipulate entire numerical arrays all at once. For example, using PDL the perl variable $a can hold a 1024x1024 floating point image, it only takes 4MB of memory to store it and expressions like $a=sqrt($a)+2 manipulate the whole image in a few seconds.

It offers both 2D [perl.org] and 3D [perl.org] visualization.

Statistical Data

[neglige]

Examples of complicated graphs include specialized bar-and-whiskers plots

I had to cope with box-and-whisker plots to visualize data gathered from a statistical survey. Frankly, Perl/Tk was nothing I looked into (and I love Perl & Tk!). I entered the data into an OpenOffice spreadsheet (which made the statistical calculation very easy!) and googled for a solution. There are many commercial packages available for Excel, but this article [qualitydigest.com] was really helpful. I managed to create a similar solution for OO (I'm not sure if the result is of interest to anyone - if so, msg me or something).

Does anyone know a repository for statistical analysis programs? Because if you only need to run an analysis once, you think twice about hacking a Perl/Tk program (the 'clean' approach) unless you are a true Perl wiz - the time needed to write the program is (with my programming abilites) two or three times longer than looking for another one shot solution (the 'practical' approach).

If there was a central repository for this, you a) knew where to look first and b) had an incentive to write the program to save someone else the time. Yes, I know I could put it up to some website, but I really doubt anyone would find it or even look for it...

much better choices around

[penguin7of9]

If you want statistically sound data analysis and graphing, look no further than the R Project [r-project.org]. It's a complete programming environment and is extensively used by working statisticians.

Python, Numerical Python, various Python plot packages, and VTK also make for very powerful visualizations if you want something more do-it-yourself.

Re:much better choices around

[holy zarquon's singi]

If you want statistically sound data analysis and graphing, look no further than the R Project. It's a complete programming environment and is extensively used by working statisticians.

I concur. I just wrote a script to do some AFLP analysis in perl (mainly for the mental excercise as a favour to someone), and aside from PDL that I don't know anything about, perl is pretty horrible for doing numerical stuff. R [r-project.org] on the other hand is pretty neat, particularly with an emacs front end. There's lots of mate

Moo

[Chacham]

Generating a visual representation is often the best way to understand large data sets,

No, it is often the *worst*. What you meant to say is that it is the *easiest*. Learn to data first, and then check what is really important. Pictures help with a quick overview, but hide the real details.