File System Round-Up Interview 112
Little Sheep writes: "An interesting round-up interview regarding modern Linux filesystems is published by OSNews, featuring the developers behind IBM's JFS, ReiserFS and SGI's XFS filesystems."
In practice, failures in system development, like unemployment in Russia, happens a lot despite official propaganda to the contrary. -- Paul Licker
Hmmm... (Score:2, Interesting)
Re:Hmmm... (Score:1)
But still, it's going to be interesting as the fs's develop and make Linux even better for enterprise level computing.
Re:Hmmm... (Score:2)
netbench recent results (Score:2, Flamebait)
http://lse.sourceforge.net/benchmarks/netbench/re
I quote
Hello all,
I recently starting doing some fs performance comparisons with Netbench
and the journal filesystems available in 2.4: Reiserfs, JFS, XFS, and
Ext3. I thought some of you may be interested in the results. Below
is the README from the http://lse.sourceforge.net. There is a kernprof
for each test, and I am working on the lockmeter stuff right now. Let
me
know if you have any comments.
Andrew Theurer
IBM LTC
seems that its all pretty much rocking and they turn out the same ish even tho they do things differant except riser which sucks and alaways has in my eyes (each to their own)
regards
john jones
Hmm... (Score:1)
Those numbers sound very suspicious. Even though the "default" config was used, I imagine that possibly the tester used a distro like RedHat which turns on debug mode, slowing down Reiser FS. When testing databases, you see shrinkwrap agreements which forbid you from publicly posting benchmarks for this reason.
He mentions he used:
linux 2.4.7, Samba 2.2.0, and NetBench 7.0.1
and it is unclear whether that was from an upgraded distro or not.
Re:netbench recent results (Score:2)
The results too perfect to believe.......may be I'm thinking too much....
the reason is small files (Score:3, Informative)
Reiser is desgned for large amounts of Dir and files which is what this tests for
and yes it achive very high marks but I have yet to see a TB on a Reiser in a live platform yet I see one every day for XFS (streaming video) and I know someone who has AIX with 3TB on it
most large files seem to be of the video nature or are database yes while MP3s are comman and thata is what Reiser did (remember mp3.com was a sponsor and came up in the boot up) but I have to say round here most peoples dir do not contain over 1000 files most have some MP3s and video, documents and the like
really I have been running XFS on a i686 for a while and have not had anything go wrong (we havent exactly pushed it tho)
what XFS and JFS need are ports to other archs and JFS seems to recogise this
remember benchmarks are written to test certain things but we commanly relie on quantum chaotic things and are unable to test for this (because they dont really have random things)
Don't forget tux3 (Score:1)
http://people.nl.linux.org/~phillips/tux2/
ext2 patch
http://people.nl.linux.org/~phillips/htree/
Tux (Score:2)
I use a nettapp all the time and a filesystem that had the same sort of funtionality would be great
I heard about this in the write up for linuxconf.au but heard little about it since
regards
john jones
http://people.nl.linux.org/~phillips/tux2/ [linux.org]
I doubt it (Score:1)
Comments... (Score:1, Insightful)
I knew the comments were getting more useless as time went on, but the majority of the first 9 posts are not only off topic, but offensive as well.
Re:Comments... (Score:2, Insightful)
Re:Comments... (Score:1)
Yeah, my threshold seems to go up by 1 every year. These days it's hard to stick with the moderating guideline to try to moderate comments up. I spend all my mod points moderating down off-topic posts.
My experience (Score:1)
I tried once ReiserFS with Mandrake 7.1. That was great and I really felt faster. :o)
But a month later when I tried to recompile my kernel I had a terrible surprise. As ReiserFS wasn't officially supported by the 2.2.x kernels I couldn't make it boot. So I had to draw back. (a month later I installed Slackware 7.1)
Well, I was a newbie then. But this bad expierence with new filesystems make me think twice before install something other than ext2, or any not-officially supported filesystem.
Re:My experience (Score:1)
And ReiserFS is officially supported now, at least for the 2.4.x kernels. It's also nice not having so see fsck take so long in the unlikely event that your machine goes down improperly.
I've only had one problem with ReiserFS. I mount my
I backed up the contents of
Re:My experience (Score:1)
Oh, I forgot to mention. I turned Slackware a year ago. :o)
Have to give credit to Hans (Score:2, Insightful)
Re:Have to give credit to Hans (Score:2)
From following the posts on the Linux Kernel mailing list, I've come away with exactly the opposite opinion. Reiser is the suit, and the others are the engineers...
BeOS' BFS uber alles! (Score:2)
In BFS, (although I'm probably going to butcher this explanation) the system actually retains the optimized parsed tree of the query, and monitors the modification times of the individual indices used in the search. When one of those times changes, the system re-queries just that branch of the tree rather than re-processing the entire query. Is very neat. Oh, yeah, and there's some notification going on.
Bah... I just really want a native file system with arbitrary indexed attributes so I can run SQL/LDAP-like queries against my non-Be machines too...
Re:BeOS' BFS uber alles! (Score:2)
It's called XFS [sgi.com]
XFS supports extended attributes just like BeFS. no wonder, since BeFS is based, at least in part, on ideas originated in XFS. You will need some userland tools to set extended attributes and probably a modified tar if you want to preserve attributes inside archives. The web page on Access Control Lists and Extended Attributes [bestbits.at] might also prove useful. Now, combined with fam & imon [sgi.com] , which provide (i)node monitoring you have everything except indexed searching, all in Open Source (imon is actually a kernel patch that adds node monitoring at the VFS layer)
-adnans
Re:BeOS' BFS uber alles! (Score:2)
Though... maybe it wouldn't be too hard to work that into XFS. Most of what I know comes from undergrad Comp Sci classes and a few readings of the BFS file systems book.
Re:BeOS' BFS uber alles! (Score:2)
I'm all for extended attributes. It makes a lot of sense to put the file system in charge of metadata. Grossly superior to using filename extensions.
Problem is, extensions are what people know, and what almost all Windows, Linux, and Unix apps are built around. Getting everybody to change won't be as hard as retiring the QWERTY keyboard, but it's up there.
And feature creep makes matters worse. Why does a filesystem need to support arbitrary relational queries? Why make every single app deal with complex access issues that only a few will ever use?
Re:BeOS' BFS uber alles! (Score:2)
Why would every single app be affected? If you don't want to use a feature don't.
On my Mac, I can search large file systems across multiple hard disks for a file with a certian name or attribute, very fast. For instance, find me all files that end in ".jpeg", are bigger than 3MB, and are in a folder that is locked. I can get results quick.
Much better than using locate/updatedb or find.
This seems like an advance to me.
BeFS and new file systems (Score:1)
Re:BeFS and new file systems (Score:2)
It tells us that it's a real neato piece of work that everyone wants, but that few could actually use, because it only ran under one OS (and it happened to be an OS that saw very little use as a server).
Filesystems, just like data formats and communications protocols, must be open in order to be generally useful.
(Novell's filesystem was able to sort of skirt this problem for a long time, because even though it was closed, it was implemented on a server, so that other OSes could "use" it without actually knowing anything about it. But they did their best to kill it anyway, by keeping NCP closed so that only a few platforms could use the server. If it weren't for sniffers, Netware would have become obsolete a lot sooner.)
Re:BeFS and new file systems (Score:1)
I dunno. I find NTFS generally useful, and have for years and years...
Re:BeFS and new file systems (Score:2)
Re:BeFS and new file systems (Score:1)
That the fact that you can not purchase a computer that has Windows plus another OS installed on it is the result of illegal contracts foisted on the computing industry by an illegal, out-of-control monopoly?
That really, really, really bad management can kill even the best technologies?
That a smart software publisher should call Palm and see about licensing the BeOS desktop to sell to the public?
Re:BeFS and new file systems (Score:1)
In fact, the BeFS's convient arbitrary meta-data attributes and indexing are a big reason that I still spend about half my time booted into BeOS. I'm really glad that ReiserFS is moving in that direction.
Re:BeFS and new file systems (Score:2)
I find it interesting that BeFS is mentioned so prominently by each of the developers as a goal for an FS to aspire to, yet the OS itself has basically died even though it was given away for free. What does this tell us?
This tells us that no matter how technically brilliant your OS is, you need developers. We went through this before with OS/2. OS/2 was much better than Windows in all sorts of ways. IBM failed to get third-party developers on board. No developers = no apps = no users. When Windows 95 came out, it sucked but was universally adopted because Microsoft was able to leverage their existing Windows developers.
Personally, I really wanted to see the BeOS succeed. It was fast, pretty, versatile, and thoroughly modern. I couldn't make the switch, though, because the apps and drivers I wanted simply didn't exist. From what I understand, this was because Be didn't work on cultivating those third-party developers that are so important.
For an interesting contrast, look at Linux. Linux is so darn developer friendly that we've got 3+ full GUI environments, 4+ journaled file systems, a bunch of web browsers, a couple of Office suites, and device support up the wazoo. We've even got some decent games (thanks, Loki!). Be didn't have any of that, despite their incredible technology. Again, developers = apps = users.
I'm really sorry to see the BeOS gone. It really was ahead of its' time. Fortunately, the whirlwind nature of Open Source development means that a lot of the good ideas in the BeOS will probably make it into another OS eventually. Good ideas never die, they just go into hibernation every now and then.
Re:blah (Score:1)
collaboration (Score:2)
Now that's cool. Projects that appear to be in direct competition, but they all have great respect for each other and actually communicate with each other. And in the end, each product ends up being better.
Corporations take note.
Re:collaboration (Score:2)
(-1 Insulting)
(+2 But it's to a big company)
(-3 They have an awesome employee server)
(+4 But they shut it down (reality.sgi.com)
Re:collaboration (Score:2)
Production Use (Score:2, Informative)
Generally, datacenter environments demand reliability over speed, and a good track record wins the day in selecting technologies.
When we implemented our database backend, we decided to go with SGI's JFS based on years of production use. So far we havn't encountered any problems!
-Marvin
Re:Production Use (Score:1)
When it comes to handling important data, he doesn't need to prove they're untrustworthy -- they need to prove they are trustworthy. The ball is in their court. By the same logic, I don't trust wu-ftpd or sendmail as secure. I may not know of any current exploits, but they have a long history to live down.
Re:Production Use (Score:1)
Pity ext3 is the de facto standard (Score:1)
Re:Pity ext3 is the de facto standard (Score:2)
Really?
I tried RH 5.2 as my first Linux. Hated it. No KDE. Other complaints.
A friend pointed me to SuSE. Boy am I glad. I might never have given Linux a second try.
I've been using SuSE for years now, completely ignoring what RH says. I'll confess I've been using KDE for several years. And ReiserFS for awhile too. But don't tell RH. Or MS.
It's nice to have choice.
Sad technical loss (Score:2)
Now, it seems pretty clear that Novell is doomed, and when it goes Netware and NDS will evaporate. I just hope that whoever turns out the lights in Provo has the foresight and generosity to release the details of those two technologies under some sort of open source license, so that even if the products disappear the technology might be saved.
But I doubt that will happen.
sPh
JFS in the kernel?!?!? (Score:2, Interesting)
ReiserFS stability (Score:1)
ease of and quickness of use (Score:1)
I think that my choice would be ReiserFS (which I've been using for about a year), because of its inclusion in the kernel.... and the way that the nice people at Mandrake Soft. have packaged it. I mean from the get-go, a fresh installation, you don't even need to have more than a 20mb /boot partition with ext2, the rest can be reiserFS.
Also, migration to ReiserFS is pretty easy... check out this link [machineofthemonth.org].Re:ease of and quickness of use (Score:1)
When SuSE first started supporting ReiserFS I installed it and it let me put everything on a Reiser partition...too bad the install only supported Reiser as a module
Re:ease of and quickness of use (Score:1)
This way you can always boot up. even if / gets fubared
Re:ease of and quickness of use (Score:1)
Re:ease of and quickness of use (Score:1)
This usually comes up with
Re:ease of and quickness of use (Score:1)
Couldn't be easier than ext3 tho. All that involves is specifying the -j switch to tune2fs.
- Arcadio
Re:ease of and quickness of use (Score:2)
JFS pulled from Mandrake 8.1 (Score:2, Interesting)
On MandrakeForum the latest news about filesystems is that JFS will be pulled from Mandrake 8.1.
There was done a test with a buildup/takedown of 100.000 files.
In the case of JFS the deleting of those files caused a hard kernel crash.
Seems there is some work to be done, despite it being a 1.0 release.
And hey, what's up with this html here?
Seems only plain text works right for me.
What about tux2? (Score:1)
The web site is here [innominate.org], but I don't really see anything giving a status update.
Anyone know anything about this project?
reiserfs on 2.4 vs 2.2 (Score:2)
I had some problems I thought were 2.4 related so I tried the 2.2.19 kernel with reiser. it didn't seem to be able to locate superblocks when it booted up and just froze ;-( so I had to go back to 2.4
this is the only downside I see. I know 2.2/2.4 reiser should work but it didn't for me. and I think I did all the right things when building the kernel AND userland tools.
I've not tried xfs and I did have some hangs with jfs when it reached 1.0.0 - but that was while on an IDE system with the infamous VIA chipset. I'd probably blame the chipset bugs before JFS, but it did shy me away from JFS.
reiser seems stable on 2.4 and I'm quite happy with it. give it a try.
Re:reiserfs on 2.4 vs 2.2 (Score:1)
Re:reiserfs on 2.4 vs 2.2 (Score:2)
now, if there was a convert util that did the re-mkfs in-place, I'd be ok with that. but I'm not about to reconvert (by hand) my entire set of reiser partitions. oh well - guess I'm stuck with 2.4 then.
Re:reiserfs on 2.4 vs 2.2 (Score:1)
I've been using a reiserfs partition interchangeably between kernels 2.2 and 2.4 with no problem at all, without the 3.5 -> 3.6 conversion.
Metadata APIs? (Score:2)
One thing that pleases me about this, is that it looks like all three of these filesystems have (or will have) metadata support. This has been a pretty serious (imho) weakness in traditional Unixes. (BTW, isn't it interesting that the platforms with the best GUIs (MacOS and OS/2 WPS) happen to depend heavily on metadata?)
I haven't used or programmed for any of these filesystems so far, though, so I'm wondering if the APIs for getting at the metadata stuff, are all the same. This is something that will be absolutely necessary before Linux app writers will be able to start using this stuff.
It looks like the teams have good attitudes toward one another. I hope they're coordinating on the APIs so it'll all be consistent.
Re:Metadata APIs? (Score:1)
A filesystem is the ONLY place I have to store information in most OS's (so, I'm discounting raw devices). It should have the best performance for the lowest common denominator. If you want complex meta data, use a 'data'base. Databases are designed for storing and searching highly structured data; filesystems should be for named unstructured data.
If you think that a database is over kill for most applications, then define a database for me. Is Oracle a DB? DBM? XML?
We do need to know what format the data is using, so the only additional peice of information that I need that is not in current metadata is the MIME type.
Joe
Re:Metadata APIs? (Score:2)
I agree that the data part of a file should not have any structure known to the filesystem. The structure of the content of a file is unstructured.
But I disagree. You said "named unstructured data". The name is metadata. There seems to be no disagreement that the name is useful. (C'mon, why can't we just all use inode numbers and do away with names?)
The metadata is to make files easier to find, and to store more information about what is in that unstructured data.
We do need to know what format the data is using, so the only additional peice of information that I need that is not in current metadata is the MIME type.
You mentioned filetype. I agree. But I don't think we should stop right there. I think "creator" is another good one. (i.e., which application will launch when I click on this file? e.g. multiple "jpeg" files might launch with different applications, and accordingly display different icons.)
Another good piece of metadata about files is their X,Y location within the window that displays them. Think about this. The Macintosh is the only OS I know of that has a geographic memory of where you put a file. I can remember, "oh, yeah, when I open that folder, I left my business plan in the mid-lower-left corner of the window."
There are other useful metadata as well.
This does not mean that existing apps have to be changed at all. On the Mac you can still write a C program that is totally ignorant of the Mac's extended capabilities. You end up with users saying "why is this program so stupid? why do all it's files have plain white icons?" But the program works. Your C program doesn't have to set the X,Y location of files, the system does that, when the user moves the icon for the file.
On Mac, every file and folder can even be given a custom icon that overrides the icon the file would display based on the combination of it's type/creator. This is another good use of metadata. In fact, it begs the question, if the fielsystem can associate metadata with a file, then why not just make the metadata completely arbitrary? The FS doesn't need to know what attributes KDE or GNOME want to tack onto a file. A window manager could tag a file as having a certian "X" attribute, and certan "Y" attribute. I custom icon attribute. etc.
But you could still type: vim mypict.jpeg, and open a jpeg file using a non-aware application.
Just because the filesystem supports extended attributes doesn't mean you have to use them. Nor do you have to run a GUI that takes advantage of them. Nor do you even have to run the particular filesystem that has extended attributes.
Some of us want better GUI's.
Re:Metadata APIs? (Score:2)
However I feel that metadata is a bad idea. The problem is that one of the main things done with a file is to copy it from one place to another. In fact with the web, I would say that non-system files are copied hundreds of times more often then they are "used".
This requires the metadata to be encoded and transmitted with the file. These encodings tend to be complex, kludgey because they are never designed for completeness, and require both ends to understand the encodign. And the end result is a stream of data that most programs (all those concerned with copying files) will treat as the file's data anyway.
There is also the annoying fact that all programming languages need functions added to them to manipulate the metadata.
So how about imbedding all that metadata into the file's data? I have thought that a system could be designed where a file is *only* a block of data. Even the file's name and protection is imbedded in the file. A filesystem would be designed so the first 1K or so of a file is very quickly accessible. There would also be simple rules so that the data could be located and it would appear as comments to whatever reads the file.
Since anybody with write access could change the permissions, permissions would actually be some and/or combination of that file and all the parent directories. This is the only complex part I see to this.
Re:Metadata APIs? (Score:2)
The very fact you might need a new API would be one major reason why metadata is a bad idea. While the concept of having metadata is not in itself bad, implementations of it that require special access are. For example, if I have a filesystem with metadata that uses special API system calls, how will I be able to backup and restore that filesystem with the metadata using ordinary tools like tar?
A simple way to have metadata is to define another file. Just append ".meta" to a filename, or replace its extension with "meta". You can now read and write as much metdata as you want to have. Then any filesystem backup or transfer tool can do the job.
Re:Metadata APIs? (Score:2)
Well, unfortunately, you can't, unless you modify tar. (This isn't all that far-fetched; I used to use an OS/2 port of tar that saved and restored extended attributes.) Of course, that only works when tar knows it's working with files, and the approach falls on its face when you're just doing streams. So yes, there's a problem.
This does mostly work, and it's what my Amiga does (.info files), and OS/2 does something a little like this when you try to use extended attributes on filesystems that don't support it (e.g. FAT). The main problem with this approach is that the files can too easily end up getting seperated from their metadata. You still need non-standard (from a Unix point of view) tools that know that when they copy a file, they have to copy the file's metadata too. "cat foo > bar" isn't going to copy foo.meta.
JFS Working Nicely... (Score:1)
Another poster commented that the source code was rough around the edges and therefore didn't merit kernel status. I disagree. The reason I don't move some of the boxes to using JFS for the root partition is precisely because it's not yet in the kernel. Thus, kernel upgrades become dicey if you can't smoothly apply the patches. Also if something goes wrong you're basically up the creek. Of course, there are backups, but that's an amazingly tedious exercise.
As for the code, well, isn't that part of the beauty of open source? At least you know what you're getting. And I've seen plenty of code - albeit not necessarily in the kernel - that looks like complete garbage but works great.
Re:JFS Working Nicely... (Score:1)
References on filesystem design? (Score:2)
Anyone know?
Re:References on filesystem design? (Score:3, Informative)
Great book, has a detailed overview of the BeOS structure, some comparisons and explanations of other filesystems (like ext2, NTFS, XFS, and HFS+), and a cool file system builder kit with full source to pretty nifty example filesystem.
Re:All journall FSs useless on Linux (Score:2)
So what is your solution?
Re:All journall FSs useless on Linux (Score:1)
Write order seems to be less of an issue there.
All your journal are belong to us (Score:2)
So when do we get to see a complete round up of all journaling filesystems (I noticed ext3 was missing), including comparisons of features (including implementation features) and benchmarks on a variety of hardware configurations (SCSI, IDE, USB, RAID)?
Re:All your journal are belong to us (Score:1)
For the record, my personal best from the 3 of them btw, is XFS.