Visualizing Complex Data Sets? 180
markmcb writes "A year ago my company began using SAP as its ERP system, and there is still a great deal of focus on cleaning up the 'master data' that ultimately drives everything the system does. The issue we face is that the master data set is gigantic and not easy to wrap one's mind around. As powerful as SAP is, I find it does little to aid with useful visualization of data. I recently employed a custom solution using Ruby and Graphviz to help build graphs of master data flow from manual extracts, but I'm wondering what other people are doing to get similar results. Have you found good out-of-the-box solutions in things like data warehouses, or is this just one of those situations where customization has to fill a gap?"
Great source of data visualization inspiration (Score:3, Interesting)
http://visualcomplexity.com
Have fun!
Re:I have a question for you (Score:3, Interesting)
the lat, lon and depth values are courtest of NOAA, freely available. this is a screenshot of a real time frame in openGL of the world with each vertex pair colored by depth. you can rotate it, probe it and a few other things.
link [rimfrost.org]
IBM data explorer (Score:3, Interesting)
Re:Edward Tufte, anyone? (Score:2, Interesting)
Tufte's ideas are good for presenting simple information. He gets many things right (eg if the visualisation doesn't work in black and white, adding colour won't fix it). However, many in the infovis community are outright sceptical, if not dismissive of his ideas for analysing high dimensional datasets.
Where his ideas really work is once you have "the answer" that you want to present to someone else. However, the basic exploration of the data to find interesting keypoints, is not what he specialises in. There's whole communities devoted to techniques for datamining and presentation, principly infovis/Visual Analytics.
Re:I have a question for you (Score:4, Interesting)
What you are doing in your post is investigating the data until you UNDERSTAND what is usefull and then presenting (visualising) it for you're boss, who probably adds another layer of "visualization" for his boss, etc. (ie: You are acting as human visualisation tool that the boss can use to visualise the output of silicon visualisation tools)
To scale up you're simple X/Y plot of two variables to corporate size you propose using a visualization tool that UNDERSTANDS database structures and UNDERSTANDS the fact that to plot strings against integers you need a default transform, etc, etc. You are handed a bunch of DB's with hundereds of tables, thousands of columns and countless transaction transforms ferrying data from one DB to the other.
So you start with all possible pairs to see if there is a nice easy curve that can relate them. You get 10,000 statistically significant relationships - the problem posed in TFS is how do you now visualize all those graphs to find the relevant relationships without UNDERSTANDING the data.
As to TFS, visualization relies on data minning which will never be "solved" because given enough data you can always add one more level of UNDERSTANDING (see: Godel [miskatonic.org]). This is not to say that trying to solve it is pointless. On the contrary, google news is excellent and accessible example of how far things have progressed in the last couple of decades.
Simply presenting multiple known facts/relationships in an easily accessible format takes a deep UNDERSTANDING of the data. Even if you do UNDERSTAND the facts/relationships, creating the format is an art that has few masters [wikipedia.org].
Re:Spotfire (Score:1, Interesting)
For those who are interested in Spotfire, here is a link to download a free (as in beer) evaluation copy of Spotfire Professional [spotfire.com], our next generation analytic client.
Tim Wormus
Analytics Evangelist, TIBCO Spotfire