'Storage' to Replace Traditional Filesystems?

JigSaw writes "OSNews is reporting on Storage, an innovative project which aims to replace the traditional hierarchical filesystems with a new document store which is database-based (PostgreSQL). The current implementation, built under Gnome 2.x for now, offers natural language access, network transparency, and a number of other features. The project is currently in alpha (screenshots already available), and it is part of the next major generation of Gnome. It is currently developed by Seth Nickell, the person responsible for the enhanced Gnome usability on 2.x and its HIG, among other things."

Windows?

[Iron Monkey543]

I thought current windows filing system was already database-based? Not that i know anythig on this matter, i just thought I read it somewhere. Can someone enlighten me please

Re:Windows?

[henbane]

Longhorn will be database based.

Why don't these people just put some effort in reiserFS?

Windows' filesystem

[mic256]

I think Longhorn will be the first Windows with a database filesystem. It will probably be based on SQL Server

Re:Ahead of the game.

[Serapth]

What an incredibly dumb thing to say... this is exactly what Longhorn is going to do, and they announced it well over a year ago!

Re:Hmm

[UnuMondo]

No, because doing away with the root filesystem, user stuff in /home, config files in /etc, and so forth would break a number of Unix standards Linux's big advantage of being able to run many Unix apps (if you compile from source) would disappear. Storage will apparently be an interface to the existing real filesystem. Joe User won't know the difference.

Re:Windows' filesystem

[cyclist1200]

The filesystem will be based on SQL Server 2003, but it won't be a fully functional version of SQL Server.

Limitations in the home edition

[yerricde]

What then happens to SQL as a MS product? If its built in to every OS, why then would anyone buy it.

Remember how Windows XP Home and Pro editions can serve files only to less than a dozen simultaneous clients? This is to boost sales of the IIS bundled with Windows 2000 Server and now Windows Server 2003. Microsoft SQL Server Home Edition will probably be limited.

AS400 did this 20 years ago:

[mikepb78]

The filesystem on AS400 is actually a db2 database and it work quite well

Not SQL Server Directly

[Watts]

Having SQL Server as the underlying filesystem technology doesn't mean that you're going to be running SQL Server directly. I mean, if you currently use NTFS, there isn't a NTFS daemon that the kernel connects to when it does filesystem transactions. Just like every other filesystem, the support will be built into the kernel. Instead of writing data as NTFS does, the structure will look a lot more like how SQL Server stores data -- with built in indexes, etc.

Many database servers already have some fairly optimized code when it comes to file access. This just implements it at the kernel level, rather than having it sit on top of a traditional fs.

BeFS

[laird]

Actually, Be had two flavors of "filesystem as database" in widespread deployment. OK, not as widespread as Windows, but certainly thousands of users. The first version of Be's filesystem, by Benoit Schillings, was very database like, but performance was so-so. The second version of BeFS, by Dominic Giampaolo, was less general in implementation, but had the same metadata-driven capabilities. There's an interesting article on this at http://www.theregus.com/content/4/24485.html. Basically, Be did everything that this project is talking about, years ago. That's not to take anything away from the project -- it's cool if more mainstream operating systems catch up to the innovations of niche players, because more people benefit. Dominic is working at Apple, so there's hope that MacOS X's filesystem will start incorporating the rich-metadata, dynamic view model of the world. And while MS has (I think) pushed the "filesystem as database" out of the next version of Windows NT/XP/whatever, it's still planned for the next version after that, so perhaps in a deade or so we'll all be able to do what Be did back in '91. And of course, Palm owns the Be code, so perhaps PalmOS will lead the way?

Re:Natural language interface? Hmm...

[tolan-b]

silly billy.

that's the sql generated by the natural language processor.

Oracel IFS

[rhinoX]

It was called IFS and Oracle did it like, almost four years ago.

Versioning and various other metadata existed. It could be exported via SMB, NFS, FTP, and as a regular "local" windows filesystem.

And, why is this such a great big deal? I don't see the same stink raised as the possibility of Longhorn having a DB for a filesystem.

Re:Why link directly againsat libpq?

[rtaylor]

Their feature list say it will work with Oracle and other SQL99 compliant databases, so I would assume it isn't linked against libpq directly.

Re:ext3 + sql

[rtaylor]

It won't improve performance if you know exactly what you are looking for. The goal is to improve performance when you only have a vague idea of what you want.

This isn't a place to store config files or cronned shell scripts which have definitive locations and content.

This is a replacement for that 5TB corporate filestore with a 50 directory hierarchy that nobody can figure out, and a content based find takes days to complete.

Re:Windows' filesystem

[Pfhreakaz0id]

My guess is it will be something like the MSDE [microsoft.com] engine. So it will be limited. For those who don't know, MSDE is just an embedded, single-user version of the SQL engine. I worked on an app once that used it for laptop users who were offline from the network and would have a copy of the database to search and enter orders in, which would auto-replicate with the master SQL server when it got back on the LAN. It was pretty neat.

Not exactly

[gilesjuk]

http://theregister.com/content/4/30670.html

"The oft-misunderstood Windows Future Storage (WinFS), which will include technology from the "Yukon" release of SQL Server, is not a file system," reports Thurrot. "Instead, WinFS is a service that runs on top of - and requires - NTFS."

Re:Obvious advantages

[Sunracer]

But there are no file extensions to rely on in the first place. When a file is first created, it will be given a MIME type when it's put to the DB. And from there, the metadata will be transferred when retrieved/copied/whatever.

going OT: library organizational schemes

[mforbes]

An OT side-note:
Just as each of us has our own organizational scheme for our own bookshelves, libraries tend to vary more than we think too.

Just about every school and community library you'll find uses Dewey Decimal, of course, but others have other schemes.
For instance: the Library of Congress, in order to conserve space on their shelves, orders their books by size. (No, I'm not kidding. Look it up.) The directory is computerized, of course, so aside from the inconvenience of having same-topic volumes wildly separated in space, it's not a big deal for them.

SQL Server Desktop Engine

[illsorted]

My guess is that they'll use MSDE, which is already freely available [microsoft.com] and "royalty free". I think it's basically just the core of SQL Server without any of the extra tools.

Re:Obvious advantages

[jeti]

You are aware that almost all internet protocols transfer a MIME-type with each file?

Re:Windows' filesystem

[deque_alpha]

uhh... he said first Windows with db filesystem, not first OS. Read more carefully before you go on crusade.

why the relational model is not right

[hansreiser]

www.namesys.com/whitepaper.html [namesys.com] describes why the relational model is not the right one for large heterogeneous stores (filesystems), and describes the approach ReiserFS (a Linux filesystem used mostly in Europe) is taking instead.

Hans

Re:Ah yes, the infamous relational filesystem...

[Anonymous Coward]

As others have pointed out, you can easily model a hierarchical system relationally, and browse it just like you do now.

But with the relational system, you could also browse it other ways. Want to implement Gelernter's Lifestreams? Just ignore the hierarchies and sort your entire filesystem by date.

You wouldn't want to force users to run sql queries, but you could easily implement more advanced views in your file manager...and make adhoc queries available for users who are up to it.

Re:so is everyone copying BeOS

[leandrod]

>

BeOS has a good idea

No!

When Codd created the relational model, there wasn't the current Unix filesystem idea... the relational model was always intended to store data, and files are data.

System R, SQL and DB2 prototype, was intended to be the basis for IBM FS.

IBM realised this in OS/400, which being proprietary hasn't the influence it deserves.

MS also wanted Jet to be the building block of its OSs since its inception, that is, sometime before MS Access release.

>

newdocms announced on Slashdot in January 2003

Sorry, but NewDocMS is based on SQLite, which is typeless and but a library... simply not good enough to be attractive. Storage is based on PostgreSQL, the real thing, and aims high.

Re:Obvious advantages

[Chelloveck]

Amen, brother! You just can't rely on metadata stored separately from the file itself. If I ZIP a file, or transfer it via XMODEM, or copy it onto an obsolete FAT-formatted floppy, that file should retain all it needs to be usable.

Some metadata is bound to be lost, such as its modification time or even its filename. If you can afford to lose this sort of metadata, then go ahead and store it separately. But if the file can't afford to lose this stuff you'd better make sure it's part of the data, not just the metadata. It'd better transfer intact when I send the file serially or copy it to and from a legacy filesystem.

Re:Journalling?

[Anonymous Coward]

Journalling prevents corruption of the metadata if you only journal the metadata. If you also journal the data, then it prevents corruption of data also. On filesystems that allow data journalling, most people don't use it, complaining that it is slow.

Comments from Seth (aka Storage's designer)

[nullity]

I suppose it is probably too late to inject comments and have them moderated to the point of visibility as the madness has largely subsided... but here's to futile acts ;-) I was not really intending Storage to make a big splash right now, I wanted to keep it low-key, but I guess the damage is done so I might as well comment. I'm sorry that I didn't have time to put up a more technically-oriented exposition of Storage. *shrug*

• Slashdot has focused almost exclusively on the "database backing". Guys, this is an implementation detail. Its an important one, but I didn't start off this design thinking "lets write a database backed filesystem store". A set of design goals was established (largely mirrored in the features page). Storage is a lot more than just a database backed XML store. Please read the features page [gnome.org]. The "searchable" stuff is nice, but equally important is providing persistent objects, uniform access (the same URI for a local storage node works globally assuming your computer has a publicly accessible IP address), an improved model for revision and "saving", the ability to localize filesystem resources, and due to a standard object format greater transparency of filesystem resources to the OS which will be useful in weakening the barrier between "apps" and "desktop" found in PCs (and not so much in, say, cell phones and pdas). This is also a key piece in an overall design of the desktop's interaction structure which I haven't had time to write up for the web.
• I'm not trying to make any claims to being the first or being highly innovative, but I am happy to make claims about improving the user experience. That said, contrary to what people are saying, to my knowledge other than the superficial layer of database backing, Storage's features do not have a "one to one" correspondence with any existing system, BFS and the only vaguely specified Windows Future Filesystem included. Most importantly these components do not seem to be a part of the same overall interaction design model that Storage is intended to support. Storage is just a stepping stone, albeit a pretty disruptive one.
• I've been quiet about this project, even inside GNOME. Storage as written today was primarily written by a team of Stanford students as their CS senior project. I've since been working with a few good GNOME developers including the person working on Medusa (Curtis) and the Epiphany maintainer (Marco). They were independently developing a metadata system for GNOME, which it looks like we may implement on top of Storage as a first major test of its capabilities. But nothing is certain right now. But the short story is that although storage is being developed by GNOME developers and I serve as usability project lead, its not an official GNOME module at this point. GNOME developers would need to corporately buy into both the Storage vision and the overall desktop design. This may never happen, and if it does, its going to be very slow in the coming.

Some technical notes... that site is sparse on technical information so I'll fill in some for the curious.

• The data store is backed by Postgresql. Postgresql rocks, though some of the features like instant notification of object changes and live queries do not fill well with existing SQL. We have ways to do all of this using Postgresql extensions, but sometimes its a little tricky and/or hackish.
• A lot of the proposed interface will rise and fall based on the quality of the NL processing. Storage is currently using some pretty cutting edge linguistics theories and tools... notably working within the basic LinGo framework [stanford.edu]. This includes using theories/systems like HPSG (Head-Phrase Structure Grammar), MRS (minimal recursion semantics), and being able to use a set of existing wide-coverage grammars such as the ERG (English Resource Gramm

Re:BeFS - further BeFS reading

[The_Blind_Priest]

If anyone would like a nice read on filesystem implementation and/or Dominic's approach to the fs redesign, check out:

Practical File System Design with the Be File System [amazon.com]
by Dominic Giampaolo, Be Inc, Dimonic Giampaolo

1. Nice overview of various filesystem's in use.
2. Quick and to the point.
3. Enough detail to go about rolling one up yourself.
4. Being written by Dominic it provides nice BeFS insights.