Haystack: A More Compelling View Of Your Data 246
Peristaltic writes "MIT's Haystack project has released the source for it's "Universal Information Client", Haystack.
In their words: 'Haystack looks into the use of artificial intelligence techniques for analyzing unstructured information and providing more accurate retrieval.' Unlike some attempts I've seen in the past to pull it all together on my desktop, Haystack shows some promise -- One of it's more useful features allows you to take the information you've been wallowing through, and have Haystack continually refine a 'dynamic hierarchy' until you get what you need. Haystack also performs some neat tricks such as combining Email, IM, web pages, etc. into a single inbox."
That's great and all, but.. (Score:4, Interesting)
It may just be me, but this is a feature I never want.
I do not want 1 large program to run all of my applications. I do not want to get my email, from where I get my web pages, and my IM. I don't want any of this.
I am quite happy with seperate programs which I can use at my pleasure. I'm happy with the lack of bulk, and the fact I can change an email client without changing a web choice. (although I only use pine anyway.)
Is this just me? Do all of you want your programs shoved together in one large application?
I didn't get any options on my cell phone (like text messaging) because I purchased a cell phone. I wanted a cell phone. To make calls. Nothing else.
Re:That's great and all, but.. (Score:5, Funny)
You mean vi (Score:2)
Emacs!
Vi!
Come on come on, one and all, joint the flamefest!
Mr. Godwin, meet the nazis of the unix world.
Re:That's great and all, but.. (Score:5, Insightful)
You mean like a Window Manager? That's how I see this thing... it's like a Window Manager with applications embedded inside of it (think of a forced dock type thing.) It just handles whatever data you present it with (or the computer presents it with) automatically.
I didn't get any options on my cell phone (like text messaging) because I purchased a cell phone. I wanted a cell phone. To make calls. Nothing else.
My cell-phone has bluetooth, PDA functions, games, voice recording, voice dialing... that's the great thing about choice. You, nor I, are the entire market.
Re:That's great and all, but.. (Score:2)
It takes different information inputs and presents them in one interface.
It doesn't appear to be as flexible though.
- Serge
Re:That's great and all, but.. (Score:2, Interesting)
Perhaps even more significantly... how does this really differ from a decent OS or Desktop-Env? I mean in GNOME I can have a Mozilla window open in a quarter of my screen, a terminal window running Mutt in another corner, OpenOffice in another part and nautilus file br
Re:That's great and all, but.. (Score:3, Insightful)
I think you are overreacting a bit. This is a research project after all so it's hardly perfect. What about your current OS, doesn't it "control all your data in a centralized place"? How is this different except for being more convenient? Do you actually care if the programs operating on your data are different processes, or plugin threads? D
Re:That's great and all, but.. (Score:2)
I like to document conversations, so I keep and file a lot of email. The IM discussions are often lost from the conceptual thread, unless I manually digest them into an email. If IM transcripts showed up with my email, it would be easier to keep a uniform record of workflow.
A more interesting solution to me would be that iChat or the like would e
Stop and think a moment (Score:2)
Re:That's great and all, but.. (Score:4, Interesting)
In my inbox I have folders for home, each client project I am working on, future leads, charitable organizations I am involved in. A similar parallel hierarchy is repeated in my file system for documents. My IM tools have their own way of tracking contacts that is unrelated to my email or projects. I store my Eclipse projects in yet another place. Mozilla organizes my bookmarks in yet another hierarchy. It's all a real mess and makes working on a project a job of mentally mapping all the pieces together.
Now, what would be real nice would be if Haystack could define a plugin API (a la Eclipse) so that my email client could be wrapped and plugged in to Haystack. Same for IM clients, web browsers, etc. The point tool then only has to worry about its job and hands off data persistence to haystack. Then I can choose the best app and let Haystack worry about tying the data together. As someone else mentioned, this sounds more like a replacement for the file system. But it could be more, if each plugin could define how it interacts with other plugins and defines its own responsibilities.
I'm sure there is a lot of refinement needed, but it is an interesting new paradigm. Activity-centred desktop insteaed of a tool-centred desktop.
Re:That's great and all, but.. (Score:5, Insightful)
So I take it you're not running Windows, Internet Explorer, and MSN Messenger?
Well, even if you're running Linux, Mozilla, and AOL Instant Messenger, they're still running on the same physical hardware and using the same window manager software in order to keep the interface consistant and organized.
And that's the point of this project and several other next-gen file systems in development now... Presenting users with a unified and organized interface that shows them their data in a way they can find it easily. From a user perspective, it makes more sense to store information as "messages that came in from Bonnie" rather than have a seperate file storage device for e-mail, IMs, voicemails, etc.
You might think it's simpler to have a physical device manage each communications protocol you use, and I'm sure product manufacturers will continue to support you with products based on that concept. However, most users would rather have their computers keep the difference between protocols to itself.
It doesn't matter how the information gets to the computer as much as what the information is and which person or organization is credited as the author. That's the best way to present information to a user who doesn't care about tech stuff.
Re:That's great and all, but.. (Score:2)
Ye olde tree-based filesystem is great, but I still believe there are better ways to organize certain information. Relational databases are also pretty popular so they must solve something the filesystem does not.
Emails are usually not files, neither are contacts or calenders. They are in files in some specific format and not accessible to all operations that
Re:That's great and all, but.. (Score:2)
# ln -s
then:
# ln -s
hows that for intuitive?
Re:That's great and all, but.. (Score:2)
If I want a gun, I'll go to a gunshop...
Oh, so you wouldn't run to a "Sporting goods" store, where you can get your gun, some ammo, maybe a camo jacket, and some wax for your skis while you are there?
Integration can get crazy - you wouldn't want to buy a gallon of milk at the sporting goods store, but in some cases (such as the Walmart Megastores) even that can make sense.
I didn't get any options on
Re:That's great and all, but.. (Score:2)
I could only hope to forget to check my voicemail, or email. That blasted voicemail light sends out a beacon to all who come by to say, "Looks like someone has a voicemail." Grrr, as if I didn't know.
I think you don't want it because you haven't seen it work well, or to your benefit.
And that's true. But let's say that it
Re:That's great and all, but.. (Score:2)
Text messaging is enabled on recieve on my phone, and people send me messages, and I have never seen a single one that was at all needed aside from, "Hey, call me in 5 minutes."
Most of them are, "R U Free 2nite?!"
Just call.
Re:That's great and all, but.. (Score:2)
Understandable, I have unlimited calling (Score:2)
I seem to forget that not everyone has that. So obviously my reasoning is different than everyone elses.
Understand your point of view very much now.
Re:That's great and all, but.. (Score:2)
That's really the only part I need to quote to sate my ego for the day.
Runtime overhead (Score:5, Informative)
From the system requirements:
- Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
- 12 megabytes of RAM (768 megabytes strongly recommended)
s/strongly recommended/REQUIRED/
Re:Runtime overhead (Score:4, Funny)
Re:Runtime overhead (Score:5, Informative)
Please take note of the following system requirements for Haystack:
* Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
* 512 megabytes of RAM (768 megabytes strongly recommended)
* Windows 2000, Windows XP, or Linux (Linux build requires GTK+ 2.0 libraries)
* At least 1 gigabyte of disk space (or more, as your repository grows)
* Java 2 Development Kit (JDK) 1.4 or later note that JDK 1.4.1 does not work with Haystack; use JDK 1.4.1_02 instead)
Re:Runtime overhead (Score:2)
Re:Runtime overhead (Score:2, Funny)
* 512 megabytes of RAM (768 megabytes strongly recommended)
* Java 2 Development Kit (JDK) 1.4 or later note that JDK 1.4.1 does not work with Haystack; use JDK 1.4.1_02 instead)
I think we've found your problem son!
Re:Runtime overhead (Score:2)
Not to mention my FreeBSD box -- it's pretty happy with 128 (yeah, that's because I don't use Gnome/KDE in case you are wondering)
Well, any new computer I'd buy today would have 512 of course, but 768???
Re:Runtime overhead (Score:2)
Re:Runtime overhead (Score:2)
- 12 megabytes of RAM (768 megabytes strongly recommended)
Thing is, the average $500 eMachines being sold today meets those requirements.
Awesome Mozilla effect. (Score:3, Funny)
Of course, IE just renders it properly. BOOOORING.
Re:Awesome Mozilla effect. (Score:3, Informative)
Re:Awesome Mozilla effect. (Score:2)
Re:Awesome Mozilla effect. (Score:4, Informative)
<div style="BACKGROUND-ATTACHMENT: fixed; BACKGROUND-IMAGE: url(images/cover.png); WIDTH: 520px; height:370px; BACKGROUND-REPEAT: no-repeat"></div>
Fun with Cascading Style Sheets :) It might've been more effective, however, to stick the big image in an iframe so people can scroll around in it easier and have a look.
Re:Awesome Mozilla effect. (Score:3, Informative)
I'm sure this isn't what the site's creator intended, as it makes it hard to look around such a pretty interface.
Re:Awesome Mozilla effect. (Score:2)
That just annoyed me. It's just a div layer clipping hack, part of the CSS2 spec. I hate myself for knowing that (I am not a web-*) but I do...
IE can't handle alpha layers in PNGs yet, I'm not holding my breath for decen
Re:Awesome Mozilla effect. (Score:4, Offtopic)
<div style="background-attachment: fixed; background-image: url(http://haystack.lcs.mit.edu/images/cover.png); width: 520px; height: 370px; background-repeat: no-repeat;"></div>
So it is supposed to be stationary. Also notice that you don't see the whole image in IE.
Whoever designed the page must be really geeky if they don't care about it working correctly in MSIE :-)
Re:Awesome Mozilla effect. (Score:2, Informative)
Re:Awesome Mozilla effect. (Score:2)
Re:Awesome Mozilla effect. (Score:2)
Except that the screen I'm looking at has an BGR striping order (while most screens have an RGB striping order) - making the image look funky.
If you're on an LCD, compare these two images of ClearType in the two striping orders (courtesy - or stolen from, or whatever, Microsoft):
THe real test. (Score:3, Funny)
Re:THe real test. (Score:2, Funny)
I'll email you my FTP address, you upload your 3 gigs, and I'll do the rest.
yeah, I promise.
Re:THe real test. (Score:3, Funny)
Yes, but that will require some optional hardware: eye-tracking camera and moisture-sensing drool-cup attachment.
Whoop de doo (Score:2)
Or maybe that's just because the A/B comparison is too easy because they're both open on my PC at work now...
I just want a relational filesystem... (Score:5, Interesting)
Group data by category, content, whatever. "Symlink" to the inodes, and you're off. We don't need AI for that and I think it would be a more complete solution. I don't see an AI engine that can correctly categorize my mp3's, I don't think I'd trust it for all of my data yet. Let's start small and get usable systems.
Spiffy program though, wish it weren't in Java... wish it weren't 42MB... wish it ran smoothly under Linux. I'll stop complaining now.
On a side note, Did anybody else find that scrolling image annoying and mentally confusing. Er, I'll really stop complaining now.
Re:I just want a relational filesystem... (Score:2)
Re:I just want a relational filesystem... (Score:2)
I don't even think you need AI. Just fuzzy logic. We already know (for the most part) the filetypes, and all standard files will have a standard filetype identifier. All you need to do is find out what type of filetype y
Re:I just want a relational filesystem... (Score:3, Interesting)
Is AI as in symbolic AI (search-based etc.)? It seems to me that all sensible information categorization systems, even those built on top of fuzzy logic could reasonably be called AI. Google uses very sophisticated data-categoriz
Re:I just want a relational filesystem... (Score:2, Interesting)
I'm using Fuzzy Logic as just a way of branching true-false trees. Not so much a full-blown AI system, just (as you said) statist
Re:I just want a relational filesystem... (Score:3, Interesting)
I might misunderstand you, but does this not mean that the "AI"-methods used by Haystack would not be AI? I also have some Neural Network / Statistical processing background so I tend to share this same view that the intelligence is in the designer not in the progra
Re:I just want a relational filesystem... (Score:2)
What I would like to have would be something that could integrate all information related to some "project" into a single, powerful interface. Something that would allow me to have information in different formats (txt, html, pdf, ps,
Re:I just want a relational filesystem... (Score:2)
The product you are looking has existed for decades -- it's called Lotus Notes, and is widely used in large companies for document tracking and project management and so on.
However, few people understand the "powerful interface", and it requires putting all your data into a proprietary database with clunky code interfaces.
Re:I just want a relational filesystem... (Score:2)
Re:I just want a relational filesystem... (Score:3, Informative)
Bayesian filtering is another of those words that mean a lot of things
Nowadays it is usually used in reference to the techniques used for spam-filtering, which is a very specific task. Classification: Spam / Not Spam. Basically everything that uses the bayesian view on statistics can be considered a bayesian method, without c
No, an arbitrary desktop menu (Score:3, Interesting)
You could have a different desktop for each project. You might have several emails for the given project, a few documents and spec sheets, some pictures, and some code. Keep the hierarchical file system underneath. Everything on the desktop is a link to something in the filesystem. Make it easy to copy, manipulate and navigate between different desktops. Basically, this would b
I don't (and you probably don't either) (Score:2)
If you want a database, just use a database. MySQL and various embedded databases are widely available on L
Re:I don't (and you probably don't either) (Score:3, Interesting)
I didn't say "relational database" -- I said "relational filesystem." As in, finding documents that are relat
Re:I just want a relational filesystem... (Score:4, Insightful)
What you mentioned is not that different from what they're doing, except they're not making it transparent -- they're making into a workspace.
I'll note also that categorization of text into topics or genres, while difficult, is easier than doing the same with music. The kinds of statistical analysis you can do on text doesn't lend itself to fourier decompisitions. To properly categorize music (in my opinion at least, which admittedly counts for little) the best technique would be to separate and identify the individual instruments (voices) in the song. This makes categorization a bit easier because now you can get data for tempo, rhythm, sohpistication of note progression, etc. on a per instrument basis. I'm not sure it's possible tho.
My 57 yen.
Re:I just want a relational filesystem... (Score:2)
To properly categorize music (in my opinion at least, which admittedly counts for little) the best technique would be to separate and identify the individual instruments (voices) in the song.
This is probably the "correct" way to categorize music, but since even humans have a hard time categorizing music I believe an automatic analyzer would have a hard time generating good results. Have you noticed that every band is X-Y with a touch of Z and W, where {X,Y,Z,W} are musi
Re:I just want a relational filesystem... (Score:2)
Because different people have very different perceptions. Of course if the artist decides it's numetal-folk-country, then he certainly is "right" about that. But then bands playing similar music might call their music neoclassical funkcore just because they have a different background and the usefulness of the classification will be a
Re:I just want a relational filesystem... (Score:3, Informative)
A file system built on an SQL engine doesn't work... It's like putting a Viper engine in a Ford Focus. A simple meta-dbm attached to each node (and visa versa, an index on the meta-dbms... simil
Re:I just want a relational filesystem... (Score:3, Insightful)
Yes, that is great news--the attempt will break so many things that it will seriously hurt Microsoft.
Anyways, in a few decades someone will write a free-as-in-no-money version for lunix. So hold tight.
DBMS-based file systems have been around for decades; there are good reasons why people aren't using them.
Linux has several file systems using database technologies (as well as change notification). However, what Linux doesn't h
Re:I just want a relational filesystem... (Score:2)
Which will of course be slower than "\\\\c\\windows\\system32\\msvcrt-7.8.29.dll". Enjoy the pig-like characteristics of your new "filesystem". ;-)
I'm pretty sure Apple or the Unix community will do a better job, long before Microsoft arrives at version 3 that actually works.
OK, that was my anti-MS rant for t
Bad news..... (Score:2, Funny)
The ultimate test (Score:2, Insightful)
The Allinwonder Pro File System? (Score:3, Interesting)
combining Email, IM, web pages, etc. into a single inbox
Whatever happened to the "does one thing, and does it very well" philosophy? If I sorta remember that I got something in an e-mail, I look in my e-mail. What's the advantage of throwing away that piece of information (where it came from)?
Yes, it's nice to use the computer to do grunt work for us, but there are some things that are better left to the user. Some of us like to come up with little "systems" for organizing things that are unique to us. We've all heard stories of the receptionist who files contacts under 'D' because new contacts are always invited for Drinks. An AI is not going to be any more rational than that, and the kooky system it devises won't be in our heads--it'll be in some obfuscated format that nobody will understand, not even the ditzy receptionist.
Re:The Allinwonder Pro File System? (Score:2)
I think that this is the main focus of the program, to improve our memory not searching all the internet, but specifically what you saw earlier and want to remember.
Of course, saying that this could
Re:The Allinwonder Pro File System? (Score:3, Interesting)
Re:The Allinwonder Pro File System? (Score:2)
Just make one of your attributes "WhereItCameFrom
If something like that is setup by default, fine. After I posted, I realized that this reminds me a lot of the DB-oriented file systems which have been discussed before on Slashdot. I think something like this would be cool at a low-level. Having a powerful database engine built into the file system is great. Having one table in a relational database is not great. In other words, this strikes me as something to be concerned about at the lower level,
So is this another search engine ? (Score:3, Interesting)
By the way Haystack people, when you use titles and phrases containing "universal", "seeks to bring [...] to the average user", "artificial intelligence" , it trips my PR bullshit meter. I was about to bail out when I noticed the download link.
Wonder if.. (Score:2)
Isn’t this named for the problem? (Score:4, Funny)
Screenshots (Score:5, Funny)
Attempting to find their server... (Score:2)
up and running is now like trying to find a needle in a haystack.
*rimshot*
Thanks /.
bah (Score:2)
But it does have a download agent, a web browser, a mail client and a newsreader all in one.
And its only a 3.7Mb download.
Hahhahaha suckers! (Score:5, Funny)
Sincerely
Bunker Hill Community College
Re:Hahhahaha suckers! (Score:2)
WTF? No Mac OS X version?!!! (Score:3)
Re:WTF? No Mac OS X version?!!! (Score:3, Funny)
Re:WTF? No Mac OS X version?!!! (Score:3, Funny)
It's assumed that if you don't run windows you are inteligent enough to organize your own info.
That would be much funnier if it didn't run on Linux.
Wait a minute .....
What's so revolutionary? (Score:2)
google cache (Score:2)
Re:google cache (Score:2)
Sorry about the first one, I'm an idiot =/
I was a usability tester... (Score:3, Informative)
I find it easy enough to edit information of the "My Documents" variety without worrying about how it is integrated into other information on my computer, and I'm sure other readers here do, as well.
The best way to actually use this software would be in the case where John Q. has a specific task to do over and over again but isn't ready to tackle a batch process.
Agents... (Score:5, Interesting)
I like the idea of bringing all my information together in one place. I don't like the idea of only having it in that one place. What I would like would be an application that can watch how I use the computer, then bring those applications together to make it more seemless.
For example, I have about four different calendars in my life: the work calendar, the one on the cell phone that I use for stuff that I can't miss, the calendar that schedules airplane rentals, and (of coursE) my girlfriend's calendar. So how do I bring those all together, and yet still be making entries in them separately?
The same is true for information. I have a primitive blogging system (really just a bunch of text files that are date coded), I have work documents that I use regularly, I have web pages that I monitor (sometimes a little too often) and I have textbooks that I'm reading (instrument flying at the moment). So how do I get all these forms of information - or at least an index into them - together in one place? But again, without changing the current organization scheme.
This is the tool that will make the computer a lot more useful - an actual organizational tool.
Rudy
Re:Agents... (Score:5, Interesting)
L.Notes had a whole wing on the now-MIA Interface Hall of Shame. It reinvents the conventions found on other platforms (it tries to be a platform unto itself) and does so badly; it's buggy, slow, and designed for administration [decent encrypted document database scheme].
Plus, it centralizes, for better or worse, all my information on servers controlled by I.T..
Now I'd love to have a central app that takes feeds from my favourite info management apps, sorts/ranks/prioritizes/interrelates the items for me according to my usage and prefs, and lets me 'zoom in' to a task by switching to the preferred stand-alone app at will. Haystack has only part of the picture, the model is still gather-control, rather than sift-sort-go.
One item I've found intriguing is StickyBrain, a sticky-on-steriods app, by Chronos LC, which takes info in many categories and allows for quick index searching, plus offers system-wide info-archiving services and some alarm and word-processing features. I had the same kind of thing running with BBEdit, a notes directory, and grep, but it was like hammering nails with a wrench.
I want all my info hotlinked to lists of related items, dynamically: make every significant word a keyword, realtime. After all, what are multi-GHz and piles'o'RAM for, anyway, when not rendering?
Re:Agents... (Score:2)
Whether or not that helps you, I dunno, but applications like that certainly do exist.
Very funny, a haystack is what I have *now*! (Score:2)
Six Degrees (Score:5, Informative)
Have you tried it? (Score:2, Interesting)
Outlook
OSAF's Chandler PIM
Haystack
Pogomail (not a PIM)
Eudora (not a PIM)
Mozilla
I am now using Mozilla because it has bayesian spam filtering built in and because it has a calendar plug in.
I have decided not to use Haystack. It is simply not production ready, and I'm sure the guys at MIT wont mind me saying so. It crashes. It lock
Scopeware Vision (Score:2, Informative)
Try the 30 day free trial. It rulz!
In the Brave New World your computer. . . (Score:2)
No more nasty "thinking." It will tell you "where you want to go today," how you want to get there, how you want to see it and how you store it for retrieval.
Choice is good, so choose not to be bothered with making your own choices. That's the ultimate choice. Brainless sheep for diversity unite!
Of course this is just a preliminary step along the way to the ultimate goal. In the Brave Newer World of the future you will turn your computer on, stick it in the closet, and it will simply do ever
Another Iteration of the OpenDoc (Score:3, Informative)
"...provides a single, uniform interface for manipulation of e-mail, instant messages, addresses, web pages, documents, news, bibliographies, annotations, music, images, etc."
"...attempts to match a user's own focus on objects in view and what can be done with them. An operation (such as spellchecking, sending an e-mail message, or rotating an image) can be invoked at any time on any object for which the operation "makes sense" (i.e. a blob of text, a person, or an image respectively)."
Back in the heady days of the PPC 601 and the Newton, one of Apple's software groups was working on this problem exactly. While I don't think OpenDoc could organize your information, it was certainly a uniform interface for manipulating stuff, with the focus on the stuff, and not the application in use. At that point, about seven years ago, I naively believed that one day OpenDoc would provide an environment in which I could edit a web page and all elements (including raster and vector images) without having six applications loaded. Ha!
Used to be called... (Score:2)
I used this product back in 1998 on my then job. I liked it pretty well then, and I understand they've made some improved versions even.
I haven't read all the threads. (Score:2)
After reading the information at their web site I think that they are trying to do the same thing that Apple is doing with tthe iApps, just that Apple is slower because they chose to integrate them differently and Apple also has to produce something that people can use now.
They want to basically integrate the different types of data that we all use on a daily basis, email, web, IM, etc in to one data repository so that any program
Re:WHAT??? (Score:2)
re: neat approach to the chaos problem (Score:2, Funny)
well, see; that was your problem right there: radix sort!
Re:Neat approach to the chaos problem. (Score:2)
Jesus christ. It's bad enough when people don't read the article, but the damn story is about the SOURCE TO HAYSTACK being released.
Maybe you can write an algorithm to make yorself more observant.
Re:Performences... (Score:3, Interesting)
It does reduce your portability somewhat of course :) I've been getting our app to run using Wine. Internet Explorer in a JVM in Wine on Linux is a bit bizarre, but we haven't seen any major speed problems with it so far.....
Re:Performences... (Score:2)
The JVM(s) on Linux are fine, so that's not it.
Sounds like a case of IE-dependence to me... :-/
Re:Haystack + ReiserFS (Score:2)
Why re-write in C? Use gcj. If the current UI stuff is Swing, either use SWT, or re-write it using something like the Lightweight Java Game Library [sourceforge.net] for fast graphics and cool special effects. :-)
Re:Haystack + ReiserFS (Score:2)
I was thinking how can such a beautiful UI be made with the crappy Swing, and I just read about some *SWT* issues with GTK+ on their site, now all makes sense...and a big light bulb is hovering over my head.
Re:The most innovative thing is... (Score:2)
Continuations in the UI are really a significant idea and have the potential to make sessions much more useful. It would also solve the problem with UI-events "stealing" the focus from what you are trying to do. Instead of popping up som idiotic dialog when one is trying to type the event would wait nicely until the user can bothered to answer the question.
I wonder if it would be p