Visualizing Complex Data Sets? 180
markmcb writes "A year ago my company began using SAP as its ERP system, and there is still a great deal of focus on cleaning up the 'master data' that ultimately drives everything the system does. The issue we face is that the master data set is gigantic and not easy to wrap one's mind around. As powerful as SAP is, I find it does little to aid with useful visualization of data. I recently employed a custom solution using Ruby and Graphviz to help build graphs of master data flow from manual extracts, but I'm wondering what other people are doing to get similar results. Have you found good out-of-the-box solutions in things like data warehouses, or is this just one of those situations where customization has to fill a gap?"
perhaps worth looking at? (Score:3, Informative)
Abstract: We propose a method for characterizing large complex networks by introducing a new matrix structure, unique for a given network, which encodes structural information; provides useful visualization, even for very large networks; and allows for rigorous statistical comparison between networks. Dynamic processes such as percolation can be visualized using animations. Applications to graph theory are discussed, as are generalizations to weighted networks, real-world network similarity testing, and applicability to the graph isomorphism problem.
PtolemyPlot (Score:2, Informative)
R language (Score:5, Informative)
Re:R language (Score:5, Informative)
http://cneurocvs.rmki.kfki.hu/igraph/ [rmki.kfki.hu]
Processing is another package that is geared towards data visualization which java developers might find easier use
http://www.processing.org/ [processing.org]
Try the InfoVis community (Score:5, Informative)
The infovis community has been dealing with these subjects for years. There's many different visualisation techniques around. Here's a list of the past conferences and the papers:
http://conferences.computer.org/Infovis/ [computer.org]
Plenty of good products out there, but the one that I like most is from Tableau Software (http://www.tableausoftware.com/).
Spotfire (Score:3, Informative)
Am I missing something or... (Score:5, Informative)
Wouldn't any everyday cube browser along with any tool to detect base dimentions in a datawarehouse schema do the trick? You may have to add a few custom dimentions on your own depending on how shitty the master data is (I don't think that can be helped, no matter the solution, if a dimention is "these two fields multiplied together times a magic number appended to the value of another table", you need to know, no tool will guess), but aside that?
Thats usually what I do anyway. I dump my data in a datawarehouse, use whatever built in wizard can auto-generate dimensions, then play with them in a cube browser. Works for even pretty archaic home-made multi-thousand-tables-without-normalization ERP systems I had to work with in the past anyhow.
Business Intelligence (Score:3, Informative)
Your ERP isn't supposed to directly analyze the data. You're supposed to use a Business Intelligence software package for that. This being SAP, I believe they'll try to sell you Hyperion.
Re:get rich slow (Score:1, Informative)
American. But I did not intend to impugn German anything.
SAP's "engineering"
There, fixed that for me.
Comment removed (Score:2, Informative)
Cytoscape (Score:5, Informative)
I had a similar situation to yours recently, except I was trying to detangle a horridly complex product substitution graph for a logistics company.
I used a bunch of Perl to crunch the raw databases into various abstract graph structures, but instead of graphviz or something created by/for developers, I found that the best software for graph visualisation is the stuff that the genetics and bio people use.
The standout for me was a program called Cytoscape [cytoscape.org] which can import enormous graph datasets and then gives you literally dozens of different automated layout algorithms to play with (most of which I'd never heard of, but it's easy to just go through them one at a time till something works)
It's got lots of plugins for talking to genetics databases and such, but if you ignore all that and use Perl/Ruby/whatever for the data production part of the problem, it's a great way to visualise it.