'Storage' to Replace Traditional Filesystems? 599
JigSaw writes "OSNews is reporting on Storage, an innovative project which aims to replace the traditional hierarchical filesystems with a new document store which is database-based (PostgreSQL). The current implementation, built under Gnome 2.x for now, offers natural language access, network transparency, and a number of other features. The project is currently in alpha (screenshots already available), and it is part of the next major generation of Gnome. It is currently developed by Seth Nickell, the person responsible for the enhanced Gnome usability on 2.x and its HIG, among other things."
i think (Score:2, Interesting)
Re:i think (Score:2)
On the surface, using a database type file system where files are just objects stored in the database along with other things seems like a great idea.
But I think that the result will probably be less resilient to damage and result in an increased possibilty of losing your data or finding them corrupted.
Re:i think (Score:5, Interesting)
One OS that solved this nicely was NewtonOS. If you wanted to manipulate persistently stored data you opened a "soup" that contained objects. So if you wanted to, say, set up an appointment with someone for lunch, you could find the person in the address book "soup" and then create an entry in the databook "soup" recording the appointment, which would immediately appear in all other apps that dealt with appointments (because app's accessed the same data structures, and were notified of changes so that they could update). So your data was not trapped in a particular application's proprietary format, and users weren't forced to learn the artificial concept of a "file" but instead could think about "my appointments" or "my address book".
If you haven't tried it, don't knock it. As a developer, and as a user, it was wonderful -- much more straightforward than "files" and "directories".
Re:i think (Score:4, Insightful)
Um, artificial as they may be, these so called "files" have been around for some time, in fact long before computers. Users can quite intuitively understand the concepts of "file" and "folder." I really think you are trying to make the difference seem greater than it actually is. (on the user side, that is)
Re:i think (Score:3, Insightful)
What programmers want is to be able to manipulate data. Period. What's so good about unix is that everything is a "file"; for instance, you can manipulate data coming from a sound card with exactly the same code you use to manipulate a sound file. You can't do this with this so-called "storage".
Re:i think (Score:4, Interesting)
1. The primary problem is implementation. Filesystems today are designed to store small numbers of very large files (ie more than 1K in size). Anybody who wants to store "objects" that are smaller than about 1K in size (like if you are implementing a "registry", for instance) is forced to write or use a database program, with needless complexity, to force all this data into a single file, so that it can be stored efficiently. What we need is a design where tiny files (like 4 bytes) can be stored efficiently.
Supposedly ReiserFS addresses this, but it is not clear if it does the necessary level of compression: ideally if you had 100 files with the same 50-bytes name and 1 byte stored in them, all those names would be in the same 50 bytes on the disk.
Sadly NOBODY seems to be trying this, and keep spouting "attributes" and "registry" and "config file". Those are all work-arounds for poor file systems.
All files must have the capability of being a "folder" and having subfiles. Any time anybody says "attributes" this should mean this sort of subfile.
2. The other problem is the blinders so people believe the "filename" is some sort of user-friendly data. This leads to brain-dead ideas like "case independence" and "wide characters" and the fact that certain bytes like "/" and zero are disallowed. This requires programs to cook data in strange ways to use it as indexes into the filesystem. This used to be true of the *data* in old systems, and we know now how horrid that was (only a rudimentary piece of that old stupidity remains in Windows text/binary distinction but I hope newer Windows systems will move that out of the kernel).
The filesystem should identify files with a counted length of bytes, just like the data in the file. In fact "name" should be a subfile of any file, and you rename it by writing a new "name". I don't think this can be solved without fixing existing filesystems.
(for "user friendly" names some form of quoting is going to be necessary. Since Windows has made "\" useless I would use that for quoting. "\0" is a null, "\\" is a backslash, "\/" is a forward slash. Just "/" itself indicates a break between hierarchy levels. For semi-Windows compatability you can also make just "\" followed by an unassigned code also mean a break between hierarchy levels.
3. The other thing that is needed (but could be done atop existing implementations) is to change the model of files. They should be "atomic" in that when you open a file for writing, you get an empty file, and this is invisible to any other program. The file only appears at the moment you close it, and only to programs that then open it for reading (programs with the same name already open continue to see the old file). Current files where you can replace a block in the middle are a special case that only a few programs use, and support can be operating-system dependent (and while you are at it, try making it so you can insert or delete data and not just overwrite).
4. As for "database" this can all be done with symbolic links (which can be implemented atop any file system which efficiently stores identical small blocks of data).
so is everyone copying BeOS (Score:4, Interesting)
Re:so is everyone copying BeOS (Score:5, Insightful)
Summary of developments:
Re:so is everyone copying BeOS (Score:3, Insightful)
Wrong! Slashdot concluded that WinFS will make computing soooo much easier that it will blow the competition out of the sky and that if Linux doesn't caught up fast it will die off.
Open your eyes people. Slashdot is not an anti-MS site anymore!!!
Re:so is everyone copying BeOS (Score:3, Informative)
No!
When Codd created the relational model, there wasn't the current Unix filesystem idea... the relational model was always intended to store data, and files are data.
System R, SQL and DB2 prototype, was intended to be the basis for IBM FS.
IBM realised this in OS/400, which being proprietary hasn't the influence it deserves.
MS also wanted Jet to be the building block of its OSs since its inception, that is, sometime before MS Access release.
Why so cynical? (Score:4, Interesting)
Re:so is everyone copying BeOS (Score:5, Interesting)
Essentially, the Be-OS filesystem, while much richer than other filesystems, is still a filesystem. This Storage thing is a full blown SQL database in the first place.
Essentially a normal filesystem is a hierarchical database where as modern databases are relational or object databases. Relational databases have proven themselves for storing complex data over the past few years.
Some scenarios to give you a clue as to why the distinction matters: you can set up a database trigger to track changes in wordprocessor documents (i.e. automatically update some table with version info whenever you click save); you can involve external databases when doing a query on your own database (e.g. the imdb example in the Storage proposal, a tv guide); emulate a hierarchical file system by associating directory attributes with an object; emulate multiple orthogonal hierarchical filesystems; integrate security policies and encryption into the database (could also be used for DRM, I know this is a sensitive topic); make the objects themselves database records (e.g. contact information); use report generators and queries to dymamically generate complex documents (e.g. software documentation, financial overviews, etc.). Use special purpose software to browse specific types of information (e.g. a picture album, movie library or an old fashioned filebrowser).
Finally something new to play with! (Score:4, Interesting)
Anything to replace NIS and its bastard stepchildren.
Replacement for ls (Score:3, Funny)
Re:Replacement for ls (Score:5, Funny)
0 rows returned
Ah, humour.
Re:Replacement for ls (Score:5, Funny)
WHERE TYPE = 'video/x-mpeg'
AND TITLE IS LIKE ('*tit*, *blonde*)
ORDER BY PERV_RANK
Screenshots ? (Score:2)
That's a good thing I think to separate filesystem and document storage. Better than vfs : either it's plain fs (simple == good for admin), or sophisticated document retrieval architecture with richer semantics than a tree (or graphs if you count links).
And then, do not let GUI apps show you the filesystem, only storage system.
Re:Screenshots ? (Score:3, Funny)
Reminds me of an internal joke we have here. Our ClearCase file server was an SGI.
Why?
Because the filenames were rendered so much prettier than on a Linux or Sun box...
Obvious advantages (Score:5, Interesting)
Of course it's only wishful thinking. I'd be nervous to see exactly how this integrates into other "Legacy" applications. I can also see be performance penalties since you are now querying a database, rather than looking at a simple file structure...
Re:Obvious advantages (Score:5, Insightful)
And how does that meta data gets to the db? Oh, right, it will rely on file extensions and other hacks
Re:Obvious advantages (Score:5, Interesting)
Like it has in MacOS for 20 years -- when applications write files, they tell the OS the filetype. The only time MacOS looks at extensions is if it's dealing with files transferred from operating systems that don't have relevant metadata. Unfortunately, that would be nearly every other OS.
Re:Obvious advantages (Score:4, Insightful)
Re:Obvious advantages (Score:3, Informative)
Amen, brother! You just can't rely on metadata stored separately from the file itself. If I ZIP a file, or transfer it via XMODEM, or copy it onto an obsolete FAT-formatted floppy, that file should retain all it needs to be usable.
Some metadata is bound to be lost, such as its modification time or even its filename. If you can afford to lose this sort of metadata, then go ahead and store it separately. But if the file can't afford to lose this stuff you'd better make sure it's part of the data, not ju
Re:Obvious advantages (Score:4, Informative)
Re:Obvious advantages (Score:4, Interesting)
Re:Obvious advantages (Score:5, Insightful)
SQL is slow compared to things like BerkeleyDB
We already have journaled file systems that can save metadata (though not user defined, I think)
Your database becomes corrupt, you lose everything.
Sorry, give me something that gives me back my data -fast-. If I want to do selects for files, I'll use locate and xargs.
--Azaroth
Nope (Score:5, Insightful)
BerkeleyDB is a hierarchial database. SQL is godzillion times faster on complex searches.
> Your database becomes corrupt, you lose everything.
Your filesystem becomes corrupt, you lose everything.
And yeah, I know about journaling, so don't bother
Re:Nope (Score:3, Interesting)
> godzillion times faster on complex searches.
Great, but who is going to often do complex enough searches for files that makes any sort of RDBMS worthwhile? The vast majority of searches would be simple keyed terms.
Re:Nope (Score:4, Interesting)
Face it: databases rock. You never know how many interesting questions you didn't ask because you couldn't think in sets until you do it, and then it's FAST as all get out.
Re:Nope (Score:3, Interesting)
> Like never?
No, like, when he suspects his system is infected with trojan or worm and he wants to get the list of executable files installed in last five days.
Re:Nope (Score:3, Insightful)
In other words, the user might never think to do that, but it'd be so cheap for the operating environment that all kinds of new applications would appear.
'jfb
Re:Obvious advantages (Score:3, Insightful)
Also, extended attributes are fundamentally broken because they're stored in the inode. They do not survive, for example, a copy operation. Worse, they do not survive an open/save cycle in most cases, because most programs do not write to open files. Instead, they open a new file under a temporary name, write the data into it, close it, unlink the original file, then rename the temporary file to the original file's name. That way
How does the metadata get into the database? (Score:5, Insightful)
Outside of a work environment, I've rarely encounter anyone who keeps consistent, useful filenames, let alone metadata indexes; it seems to me that people will skimp on the metadata, and thus limit the usefulness to metadata that the computer can collect automatically ("All movies that last under 90 minutes"). It's like CD collections, or books; libraries have nicely catalogued and ordered collections. Private individuals don't; they have roughly ordered collections on the shelf, and don't bother keeping them in any better order. I suspect the same will happen with these metadata systems; people won't do the work needed to make them truely useful.
Re:How does the metadata get into the database? (Score:5, Funny)
Call yourself a geek? How can you possibly but something on a shelf without first checking to see that it's in a proper place observing the subtle cross reference system that backs up the obvious system. Man, I hate it when people move my stuff.
Re:How does the metadata get into the database? (Score:2)
-Rusty
Re:How does the metadata get into the database? (Score:3, Funny)
Geez, do you have to count how many times you wash your hands? Do you make sure your right turns always equal your left turns?
Try this:
Leave a big cardboard box on the floor. Dump all your CDs into it. When you want to listen to music, fish around and enjoy the surprise at what you find.
Enjoy the extra free time you have now that you are not obsessively organizing your collection.
Re:How does the metadata get into the database? (Score:3, Insightful)
Re:How does the metadata get into the database? (Score:5, Insightful)
Other metadata is automatically inserted. When you install OpenOffice, it asks for your name and inserts that as the author into any new documents you create, for example.
Sure, the metadata on my personal machine will never be comparable to what a library could do. But it doesn't have to be - it has to be useful for me, not - like the library - for thousands of people with very different interests and approaches.
Re:How does the metadata get into the database? (Score:4, Insightful)
Some intrinsic metadata can be extracted with automation. For example, it's pretty trivial to examine a TIFF and tell you that it's X by Y pixels in a given color space. It's harder, but still possible, to tell you that the TIFF is predominantly red and green. It's impossible for the computer to tell you that it's a picture of a barn.
The same is true of extrinsic metadata: some of it can be extracted automatically, but not much. An example of extrinsic metadata would be a relationship. The computer can tell you that main.c and somefunction.c are both C language source code files, but it may or may not be able to tell you that they're both part of the same program. If the two files are explicitly related to each other through a makefile or some such, then the computer can know that they're related. But consider the collection of JPEGs I just copied to my home directory from my digital camera. A dozen of those pictures were taken in Fiji. The computer cannot know this unless I tell it, nor can it know which pictures were taken in Fiji and which were taken in my back yard last Tuesday. Thanks to my camera, the computer can know what apeture and focal length were used for each picture. In theory, if my camera had a GPS receiver in it, the computer might even be able to tell me there, on earth, the camera was when it snapped a given photo. But these are just automatic methods of telling the computer about the pictures. They're conceptually no different from sitting down and typing the information in. The point remains that the computer can figure out some things on its own, but it cannot know most things unless it is told.
You don't have to strain your imagination to think about this stuff. Consider your MP3 collection. Your computer can tell you that a given MP3 is 6:15 long, and that it's 192 kbps, and that it's stereo. It can't know that it's "Treefingers" by Radiohead unless somebody tells it first.
So you're basically right: automatically extracted metadata is marginally useful, but the really useful stuff has to be manually entered. And generally speaking, even in business environments, that sort of information simply never makes it into the database. It exists exclusively in people's heads.
That's the biggest challenge of digital asset management--which is, incidentally, essentially what we're talking about here. The biggest challenge is how to take information that people have in their heads and store it in some structured, persistent form. That form might be a three-ring binder or a card catalog or an Oracle database; the challenge is the same even if the technology is different.
Bottom line: this technology is really neat, and has limited applications in which it's very useful. But it is not generally useful, nor does it have widespread applications.
Re:How does the metadata get into the database? (Score:5, Insightful)
Another argument to prove you right is the "rate this song" options in iTunes. With that feature, one can assign 1 to 5 stars to a song so that later, you can quickly select your favourites. Such system is flawed in 2 ways
- I have yet to encounter anyone who uses it exhaustively. Most folks rate a few dozen songs. I have a library with 9000 mp3s and sure as hell I'm not going to spend a whole week rating them.
- I have yet to encounter someone who uses it consistently. Today I might consider Chris Isaak a 4star song since I'm in a depressed mood and it's raining outside. Tomorrow the weather might be beautifull and I mod him down to 2 stars cause he's a bloody negative wanker.
This is offcourse iTunes specific, but it shows that the assignment of metadata is far far far more complex than the methods to search/organize the stuff, which is what the "Storage" software above is about.
As an extra complication : consider that my metadata might not match someone elses. For instance if I were to label a mail "message from my brother", the same content would be "message from my son" for my father !
The fact that metadata based filesystems are not on our desktop is perhaps more to the fact that it's not a valid solution for data on desktop computers. Maybe, just maybe this is not due to MS squashing Be, as someone else was karmawhoring above.
Re:How does the metadata get into the database? (Score:3, Insightful)
Granted the end user is not going to be likely to maintain this information, but that doesn't really matter. The end user is also not likely the source of the material in the first place. I contend that metadata is more useful for material that the user has down
Re:How does the metadata get into the database? (Score:3, Interesting)
If it allowed natural language interaction with the machine, people might just provide more information. Since it begs for a voice interface, why not have the machine ask a few questions about a document while you're editing/viewing it? When a new file comes in via email with no metadata, the machine says "what's this all about?". You'll naturally describe it using words similar to those you'll use to retreive
ext3 + sql (Score:2, Interesting)
What can really interest me is something like updatedb/locate but with SQL syntax support, this could be awesome.
Re:ext3 + sql (Score:5, Informative)
This isn't a place to store config files or cronned shell scripts which have definitive locations and content.
This is a replacement for that 5TB corporate filestore with a 50 directory hierarchy that nobody can figure out, and a content based find takes days to complete.
Re:ext3 + sql (Score:3, Insightful)
It also solves the problems of if I have a letter to Smith and Sons (Builders) about a bridge construction project do I store it under letters/Smith&Sons/Bridge or Smith&Sons/letters/bridge or bridge/Smith&Sons/letters or projects/builders/bridge/smith&sons/letters or what. With a database based file system you store it once and tag it as a letter, to Smith and Sons (Builders) about the bridge construction project (and any other tags you would like to apply to it) and then when you want to
AS400 did this 20 years ago: (Score:5, Informative)
Natural language interface? Hmm... (Score:2)
"select DISTINCT(recordid) from AttrSoup...."
Well call me old fashioned , but in my day we called that SQL. Why do I get the feeling this is
just yet another database dressed up to look like its providing some always-wanted-but-until-now-folks-it-just-wasnt-p o ssible-but-with-new-WizzoFS- etc etc
Call me a cynic but I've seen it all before. Besides , databases are inefficient for manipulation filesystems at a low level so expect
your PC to crawl if you use this on
BeFS hello?? (Score:2)
This is essentialy what Longhorn's taked on SQL extensions are going to provide, and I had no idea there was ongoing progress to have this functional in *nix so soon! By the time 2005 rolls around, I have a feeling this will be a lot further a long than microsoft's implementation.
*cracks whip* On zerocool, on uberh4ck3r, on coding monk
I18n? (Score:2)
Storage (Score:2)
I have a new way to get between point A and B. I call this product "Car". To fuel it, I've started a fuel company called "Gas". Of course, people will abuse "Car", so I've also created something to keep them in line, called "Fuzz". Fuzz will be powered by what I call "Donut".
(hey, it's Friday, gimme a break :-)
Patents? (Score:2)
Why link directly againsat libpq? (Score:3, Insightful)
Re:Why link directly againsat libpq? (Score:4, Informative)
ok, (Score:2)
I spot a pirate! (Score:2)
As for the idea of database file systems -- I don't think we need this yes. Both file systems and database research should concentrate on distributed
ReiserFS future (Score:2)
Multiple paths to the same object seems perfect to me.
"Damn, I left that on my roommate's desk" (Score:5, Insightful)
"Stanford."
"No problemo, I'm heading that way later and I can grab it for you. What's your room?"
"Dorm 5, Room 109. It's the desk on the left."
( We didn't bother to state earth.us because we were already inside those directories)
Yes, yes we do think heirarchically. Most of the history of human thought has been fitting everything we can lay our filthy little brain cells on into heirarcheis, whether they wish to fit into them or not. It's intuitive.
As for natural language didn't we learn about that with COBOL? Natural language only speeds the learning process slightly ( the majority of the learning still lying in the realm of understanding the basic concepts involved), but then becomes a pain in the ass forever afterward.
Looking at the screenshots it's also ugly as all sin. The physicist in me can't help but feel that a model that ugly can't possibly be correct.
I think this makes just about as much sense as using a document preperation language (XML) as the basis of a database.
Which is to say, none.
KFG
Re:"Damn, I left that on my roommate's desk" (Score:4, Interesting)
I think the examples he shows are pretty good. In my mp3 collection, I would like to see "All bluegrass songs" or "all remixes of Parliament Funkadelic stuff". How do you propose to do this in a hierarchical filesystem? Most of my bluegrass artists are under 'bluegrass', but then there are some bluegrass songs that were in non-bluegrass artists and albums folders.
In my workplace we are having the same problems. On our shared folders, we have shipping documents in each clients' folder. But then, what if we what to see all shipping documents from a particular vendor? Currently, we would have to go into each customers' folder (which are also broken down by year archives) and grab all documents which *might* be from said supplier, and then open each one, and look to see, because the supplier name isn't in the filename. It's horribly broken, which is why we are moving to a database storage system for such documents.
Random thought for the day... (Score:5, Funny)
Granted, it's mostly pr0n on there, so it's almost the same thing, but still...
Oracel IFS (Score:4, Informative)
Versioning and various other metadata existed. It could be exported via SMB, NFS, FTP, and as a regular "local" windows filesystem.
And, why is this such a great big deal? I don't see the same stink raised as the possibility of Longhorn having a DB for a filesystem.
Microsoft Attempts for decade,GNOME Does in months (Score:5, Interesting)
1995 Signs to Cairo [byte.com]
1996 Unearthing Cairo [byte.com]
The so call Longhorn WinFS directory is just another rencarnation of the Cairo object orientated file system.September 1, 2003 Eweek 'Longhorn' Rollout Slips [eweek.com]
Microsoft have been attempting this type of functionality since 1991, over a decade. Meanwhile, one open source GNOME developer, with help from the other core GNOME developers, provides most of the features within months [gnome.org].Re:Microsoft Attempts for decade,GNOME Does in mon (Score:3, Insightful)
So, I think this GNOME thing will also sizzle out after a while.
GREAT! If it is done well... (Score:5, Interesting)
On my home systems, I have over 250GB online. That doesn't even count my music or videos/movies, which I keep on seperate, removable, optical storage.
I can tell you from experience, that managing that much data is a huge hassle. Let's say you've got your files organized well. You probably have hundreds of folders for each subject, and you have to broswe to each one with each new file you save. I have a folder (several actually, for various subjects) where I save thing that I've haven't taken a look at yet. Let's say it's a program that I haven't installed. Well once I do install it, I need to clean up all the temporary files, then browse around to another folder (takes a minute or two when you have hundreds of folders), where I save installed programs, and browse to the appropriate sub-folder, and save it. But then I end up doing the same thing with a video clip... Watching it, deciding to delete or save it, then browsing to a sub-sub-sub-sub folder to move it.
Of course, that's enough of a hassle, but things get complicated when I want to move things to another systems, which obviously isn't going to have the same filesystem. Merging each individual folder, into each different folder is seriously time-consuming, and teedious. Without fail, there always ends up being a couple folders in the wrong place, because they were a sub-folder of something else, that I did happen to see when I coppied the contents of the folder.
Then matters are even further complicated, because I may choose to delete older content months later or so, and locating everything is a huge mess.
Personally, I would like to save everything in one place, not having to change folder to folder for each file. When saving something, I could just enter a handful of keywords (eg. "picture penguin snow") which would be much less work than moving to directories or even typing in a long filename. From there, a simple database system would be be able to know what type of file it is, how large it is, and how old it is. That would make it incredibly easy to manage. Whenever I want a file, I type-in "images older than 1 years" or "programs marked as archived" and I get EVERYTHING I'm looking for in a fraction of the time. Not only that, but it makes pruning out old data as easy as it could possibly be. Just search for "linux" and delete older version, no worries about what folder it's in... If it's in a temporary folder and you haven't used it yet, or if it's archived and been in-use on your system forever. Obviously you'll be able to see that information, but unlike in our current systems, it won't stand in your way when you want to find things.
It's absoultely no work at all to transfer files, since the info should stay with them, and it will automatically integrate perfectly with your local file management/organization scheme. What's more, data like marking something as "archived" is great in that your system could automatically move it over the network where you archive your files. Since your filesystem would be a smart database, when you search for the file, it could still turn up in the search results, and be automatically moved back where you need it, when you need it.
Personally, I think this would not only save time and effort, but money as well, because so many people wouldn't be dealing with their file problems by just throwing more space in their systems, instead of spending time on figuring out where every file is, what they can get rid of, dupilcate files, and junk like that.
With this, I should be able to say "tar -xjf 'newest version of mplayer'" However, this will need to be in the actual filesystem to be useful, not just supported for GNOME applications.
Re:GREAT! If it is done well... (Score:3)
You'll have to type "I do not want a bigger penis" to remove them all.
Heh.
Ask a librarian how (Score:3, Insightful)
Librarians have been working on these problems for centuries, why not start with what they know?
Ah yes, the infamous relational filesystem... (Score:5, Interesting)
The relational zealots are quick to point out that a relational system can model any sort of data. Indeed, it can do this. This does not, however, mean that it's always good at doing this. Sometimes it's the right tool for the job, and sometimes it's not. In this case, it is very much not a good tool for sole access to files on the system (though it can make an excellent tool for complementary methods of access).
The reason that hierarchical filesystems have survived for so long is due to one thing: navigability. It's relatively easy for any user to browse what's on the system and get a good idea of how it is organized.
You can't navigate a relational system, which will prove to be the downfall of any all-relational system which comes into being. You can, of course, do a SELECT * FROM volume if you really want to, but that does exactly that: it gives you all the data, with no particular organization. Examining the entire "sea of data" suddenly becomes cumbersome in the extreme. So while User A might be able to set up an all-relational filesystem completely according to his own tastes, User B will be totally lost on that same system. This is, to say the least, a nightmare for anyone working in a shared environment.
This is not to say that the relational model isn't necessarily a useful thing for filesystems. On the contrary, it can be very useful for certain kinds of searches. As time goes on, I believe we'll see more relational-style searching technology incorporated into file managers and search tools. However, there also needs to be a means of hierarchical navigation. Humans tend to think of things in terms of locus, and a means of providing that kind of reference point have to be maintained.
Luckily, this can actually still be emulated using relational-style tables, even though it's somewhat less efficient than classical storage techniques. Some filesystems already do something similar to this, and the results are promising. Look at Be's filesystem for an example of that.
The best way to go, moving forward, is something not unlike what BeOS did, with both hierarchical and relational methods of examining data. This allowed for the best of both worlds. The default method of getting at data is still the hierarchical paradigm, but relational searches can be applied to create what some have called "smart folders" (perhaps "boxes" might be a better term?) Systems like this "Storage" should be focusing on complementing traditional systems in this way, rather than replacing them.
No, no, no!!! (Score:3, Interesting)
No, this isn't what is needed. Hierarchical object-oriented persistent object trees is what is needed.
Let me explain.
Information in real life is organized in trees. It is obvious anywhere one can look. From the organizational chart of a company to the chair that you are sitting on, everything is a tree; a tree of information, where each little piece of information consists of other pieces of information.
If you check computer applications, almost all application contains some sort of tree. Take a Word document, for example: the master document, the contents, the heading 1 and subheading paragraphs, the pictures, the drawings. Everything is a tree, and the document can be browsed as a tree.
Take your favorite mail client. Information is organized in a tree: inbox, outbox, sent, trash; each of these contain an e-mail. An e-mail itself is composed of subject, body, attachments. The body consists of paragraphs; the attachment consists of files.
Take your favorite drawing program: the picture consists of layers; each layer consists of shapes; each shape may consist of other shapes.
Take your favorite 3d/cad program: a 3d object consists of other 3d objects.
Take the gui: a window consists of other windows.
Relational databases don't provide tree organization. I don't want a freaking flat table to store my documents. I want to organize them in trees. That's why a filesystem has subdirectories.
The biggest problem of the current filesystem technology, is that a 'file' is as dumb as it gets: it is just a collection of bytes, waiting to be manipulated by some other program. It is even untyped, for God's shake!!! one program may view it as a series of bytes, another program may view it as text, another program may view it as code!!! The file itself can't tell you anything about its properties, about its contents, about the way it is supposed to function....
If a file could tell the outside world how to be operated, then the world would be a much better place. If a file could tell me its properties, if it provided me with the tools to manipulate it, then I would make any type of app that processes the file as needed.
The above is essentially object orientation on the filesystem level. RDBMS don't offer such kind of functionality!!! at best, an RDMBS offers an index on a key for quick searching, and that's it!!! There is no notion of tree, nor each file exposes its properties/methods/functionality to its users!!!
So I say a big 'NO' to relational filesystems.We can immediately move to the upper level:
1) each node of information is AN OBJECT.
2) the object specification is defined at the filesystem level. Much like COM or .NET or CORBA.
3) each object can contain other objects, if it inherits and implements a specific interface.
4) each object is PERSISTENT. The filesystem takes care of persistence, according to attributes of the object's fields. Complex objects that are composed of other objects are also managed in the same way.
5) the parent object provides the storage implementation. The storage implementation would be object-oriented!!! An object could implement an RDBMS-like storage capability with indexes, keys, etc.
At each given time, the information model inside the computer could be:
1) splitted in multiple computers.
2) shared by multiple users.
3) checked for security in ONE place, inside the operating system.
4) provided as a framework to programming languages.
5) replicated across sites with minimum of code
6) a unified GUI could handle it
7) searching through it will become a breeze!!! (for example show me all MP3 with artist = Elvis and title = rock)
After all, 90% of the programming goes to load/store and display information. It is silly not to provide a unified mechanism for that. And a simple SQL-based RDBMS does not cut it.
why the relational model is not right (Score:5, Informative)
Hans
Voice recognition? (Score:3, Interesting)
as long as it stays in user space (Score:4, Insightful)
But "replacing the traditional file system" carries with it the notion of ripping ext3 out of the kernel and putting a relational database there. That's a very bad idea. Databases don't belong into the kernel. They are far too inefficient to handle most storage needs, they are far too complex to go into the kernel, and they just don't need to be in the kernel. Operating system kernels need simple, fast storage systems. Something like ext3. ReiserFS is pushing the limits. PostgreSQL would be going too far.
As an aside, this is an idea that just about every nerd has when they learn about databases and retrieval. It's been tried various times since the 1960's. There are probably good reasons why interfaces don't use them. Perhaps most importantly, keep in mind that the vast majority of files on your system are not user files, they are bits and pieces of the operating system. And for the files that actually are used by users (mail, PIM info, images, text, etc.), they usually already have special-purpose database interfaces available to them as part of the applications that users use to access them.
Only as good as your data (Score:5, Interesting)
Where does this metadata come from? I assume I have to enter it myself. This means the more files I have, the more detailed and specific my data entry becomes. And that much more tedious.
Even worse is the uncertainty that would arise. Is my search for "horn solos" not returning results because there are no such files, or because the filesystem does not have meta data describing the files I want as such?
At this point, hierarchial organization once again becomes much more appealing again.
Comments from Seth (aka Storage's designer) (Score:5, Informative)
I suppose it is probably too late to inject comments and have them moderated to the point of visibility as the madness has largely subsided... but here's to futile acts ;-) I was not really intending Storage to make a big splash right now, I wanted to keep it low-key, but I guess the damage is done so I might as well comment. I'm sorry that I didn't have time to put up a more technically-oriented exposition of Storage. *shrug*
Some technical notes... that site is sparse on technical information so I'll fill in some for the curious.
A filesystem *IS* a database! (Score:3, Insightful)
A filesystem is a special case of a database. So it is perfectly acceptable to store your data into a filesystem. Some people seem to think everything has to be put into a relational database or that is it somehow cool to do so. I have seen people store loads of graphics as BLOBS in databases. Someone once suggested storing a ton of MP3's in a database. Most recently someone said (and this isn't the first time) that we should store all of the emails in a database. It's just another unnecessary layer of complication, especially when you are going to be referencing the email/graphic/mp3 by name all the time anyway (fs's like reiserfs index on name so it's blazingly fast) and not by a bunch of other pieces of meta-data. And if you are going to need to do lookups by various bits of meta-data then store the meta-data in a db and also store a record pointing to the actual file on disk. I have done that lots of times and it works great.
Re:Windows? (Score:2, Informative)
Why don't these people just put some effort in reiserFS?
Not exactly (Score:5, Informative)
"The oft-misunderstood Windows Future Storage (WinFS), which will include technology from the "Yukon" release of SQL Server, is not a file system," reports Thurrot. "Instead, WinFS is a service that runs on top of - and requires - NTFS."
Re:Windows? (Score:3, Funny)
Windows' filesystem (Score:3, Informative)
Re:Windows' filesystem (Score:2, Interesting)
What then happens to SQL as a MS product? If its built in to every OS, why then would anyone buy it. Ive seen MS build o
Re:Windows' filesystem (Score:4, Informative)
Re:Windows' filesystem (Score:5, Funny)
you mean it will be a standard version of SQL server?
Limitations in the home edition (Score:5, Informative)
What then happens to SQL as a MS product? If its built in to every OS, why then would anyone buy it.
Remember how Windows XP Home and Pro editions can serve files only to less than a dozen simultaneous clients? This is to boost sales of the IIS bundled with Windows 2000 Server and now Windows Server 2003. Microsoft SQL Server Home Edition will probably be limited.
SQL Server Desktop Engine (Score:3, Informative)
Not SQL Server Directly (Score:5, Informative)
Many database servers already have some fairly optimized code when it comes to file access. This just implements it at the kernel level, rather than having it sit on top of a traditional fs.
Re:Windows' filesystem (Score:5, Informative)
Re:Windows' filesystem (Score:3, Interesting)
Re:Windows' filesystem (Score:5, Interesting)
The idea was probably stolen from Xerox Parc in the first place, of course.
Re:Windows' filesystem (Score:4, Insightful)
I do deserve my wrists slapping though... I'd completely forgotten about BeOS! For shame!
Re:Windows? (Score:5, Insightful)
There are also issues with gaining acceptance for the change in the way things work. This kind of thing has not really been done on a large scale in the wild before, on any OS, so whether people will be willing to accept the security and reliablity issues that may ensue is another matter. For example, what are the implications of a compromise in the database engine? MS is planning on using SQL, so if things go awry and it becomes possible to maliciously inject raw SQL to the filesystem interface... Oops. On the otherhand, the benefits for data retrival are *huge*. Imagine being able to find any audio files on your entire system by Justin Timberlake or Britney Spears and delete them all in one go by searching on the tag fields! ;)
(1) Technically, all filesystems are databases, it's just that current ones are a collection of flatfile database tables that can point to each other, generally in a heirarchial manner. When people say "database" in the same sentence as "filesystem" they usually mean "relational database". As an aside however, high end databases usually forgo the need for a file system and provide the ability to write their tables directly to disk on a dedicated partition.
BeFS (Score:4, Informative)
Re:BeFS (Score:3, Insightful)
there's hope that MacOS X's filesystem will start incorporating the rich-metadata, dynamic view model of the world.
you mean like Mac OS 9 and earlier?
Re:Windows? (Score:3, Insightful)
Actually, I was just thinking about this problem, and you know what would make a *really* easy solution and is readily available already? P2P! Think about it; a new file arrives on the system by whatever means, so the file system has zero idea about it's nature beyond what's available from the file. We probably know the type of file from its header, extension or whatever o
Re:Windows? (Score:3, Insightful)
Re:Windows? (Score:3, Funny)
Re:Ahead of the game. (Score:3, Insightful)
Yeah, and as Longdong gets pushed back and delayed and delayed and pushed back and postponed and delayed, it'll be last to market but microsoft will still have been the first to announce it. I guess that's more innovative than they've been in the past when they'd simply wait for someone to do something interesting before buying them out.
It's not enough to say. One has to do. Microsoft has proved many times over that it often makes grand announcements only to provide something far more watered down by the t
Re:Ahead of the game. (Score:2, Interesting)
This, IMHO, is a good thing. The big difference between MS and Open Source on something like this... in Open Source land, you can often see progress from day one... no matter how unstable it is. With MS, you wont see anything until the whole product is done... Not saying one is better then the other, but...
Re:Hmm (Score:4, Informative)
Re:filesystem is a database (Score:4, Insightful)
They are original ideas, they just don't make it into the PC world where MS dominates. MS come up with as many original ideas as McDonalds
and since all KDE & Gnome (and frankly most open source projects) are doing is playing catchup with MS then originality is never going to be
a prime concern.
Re:Backups (Score:2)
Re:Backups (Score:2)
It sometimes dumps database objects in the wrong order, and restore fails as a consequence.
Re:woot woot (Score:2)