Toward a New Kind of Linux Distribution

An anonymous reader writes "Progeny co-founder Ian Murdock wrote a weblog entry that has been reprinted at Newsforge. He talks about how current distros are built from the top down, making a 'one-size-fits-all' solution of technology. He proposes making a modular solution that encompasses building modules so distros can include only the technology they need to suit their purpose, kinda like building from the bottom up. Interesting read, good arguments, potential for a new Linux community."

Ian

Progeny co-founder Ian Murdock wrote

Wouldn't it be worth mentioning that he is founder of Debian as well?

Re:Ian

And the name "Debian" is a contraction of his name (IAN) and his "ex-girlfriend" name (Deborah) = Deb+Ian.

I notice the debian graphic...

but wouldn't something like this be better based on Gentoo? If it's going to be modular and simple to use for the majority I think it'd be better off with package management more along the gentoo line, instead of debian, which while good is more suited to hackish, more finely grained options?

why?

Never mind that Ian Murdock is also a founder of Debian, and that Progeny has always been built on Debian; what objective reason is there for building this kind of OS on Gentoo rather than Debian?

First of all, Debian is quite modular and simple. In fact, Lindows uses it behind their "click 'n' run" front end, and its supposed to be amazingly smooth. Debian can be used for more finely grained options, but can also be used for a modular system as described Murdock.

Plus, lets be honest; source distributions just aren't going to cut it in an environment where package installation speed is important.

Re:why?

Hint: update to something made within the last year and there's no difference between installing/updating gentoo and any other OS, and you get the advantage of gentoo's optimization in space and speed. I know which I'd go for.

I installed debian with a full kit of KDE in under an hour on an iMac 400. You're saying there's some new hardware that can build that equivalent with gentoo within 60 minutes?

Bullshit.

Re:why?

I installed debian with a full kit of KDE in under an hour on an iMac 400. You're saying there's some new hardware that can build that equivalent with gentoo within 60 minutes?

Just tell emerge to use precompiled binaries instead of compiling everything from source, if that's not your thing ;)

Re:why?

Do you know of any Gentoo precompiled binaries repositories? The binaries that come with the 1.4 Live-CD are now all outdated, and looking in Google only brings some binaries...

Wait, I just looked again, and it looks like there's one server that supplies binaries for the different CPUs... sweet.

Re: meta-packages

Every package management system which supports dependencies (apt, portage, ports, etc etc) will do this. KDE is just a package which depends on all those other packages. In fact this is how it works in the real world anyway: A full KDE3.x environment is made up of such and such libraries, such and such applications, and so on. They just release a new ebuild for the new KDE version, which depends on all the new versions of the component pieces, and when you emerge -u kde all of those packages will be upgraded in order, because the system tracks dependencies. It handles it in much the same way as debian, actually, in that you can make things depend on and provide virtual/foo (for instance vim supplies virtual/editor.) You can also have packages depend on actual packages, with specified version numbers, or you can specify a version >=, etc. Of course you probably know this stuff, but I felt it was worth mentioning here. Anyway apt provides dependencies so debian has the same feature.

Re:why?

Hint: update to something made within the last year and there's no difference between installing/updating gentoo and any other OS, and you get the advantage of gentoo's optimization in space and speed. I know which I'd go for.

Yeah, that 7+ hours compiling KDE/GNOME just flies by

Impossible

Hmm, well, it seems you might be slow, so I'm going to state this just about as clearly as I can.

It's impossible for Gentoo to be as fast as binary-only distributions because it has to the job of the binary distribution (the "make install" part) in addition to the compilation. Which, by the way, is slow (with any program or reasable size) on any hardware. I do use an athlon XP 1600+ which is fairly old (and did indeed perform quite poorly at installing Gentoo packages), but even on a Dual Xeon system, I wouldn't want to have to compile KDE from source.

But the most important thing to note is that many people do use old hardware. Why not support them as well? My work computer is a P3 700, and it runs Debian quite smoothly, and installs even big packages in less than a minute (of course, it helps that my work connection gets > 1megabyte/second to MIT's Debian mirrors). Why should that hardware not be viable? Just because you think everybody should use source only distributions? I don't think so.

Re:Impossible

You're talking about install time. Yes Gentoo is 'slower' at installing apps. However, once the app is installed, it is much faster than the generic i386 cruft you get from normal distributions, simply because Gentoo apps are compiled for your specific processor.

Which do you do more over the life of an application; install it or use it?

The obvious answer is that you use it far more times than you install it (only once per version). Gentoo's method clearly wins in that respect.

But the article is about modular distros, not performance. Gentoo is pretty good at that as well. It will get better at it too. I don't have a link handy, but you'll be able to customize your own Gentoo Live CD (a.k.a Modular Gentoo).

Re:Impossible

"much" faster? Only the kernel and glibc make more than a negligible difference; Fedora, Mandrake and other distros offer CPU-specific versions of those.

Equally, such flags as -O3, which Gentoo users seem to apply to everything, can actually be WORSE as they increase code size (function inlining) and thus result in more CPU cache misses.

In short, it's not black-and-white, and if you do some proper testing Gentoo is NOT "much faster" at all. In some cases it's slower. But most of the time the differences are barely noticable.

Re:Impossible

Equally, such flags as -O3, which Gentoo users seem to apply to everything, can actually be WORSE as they increase code size (function inlining) and thus result in more CPU cache misses.

I'll admit that my knowledge of processor architecture is quite limited, but it seems to me that inlining functions would prevent cache misses since you're just running a long sequential block of code instead of jumping around to run code from different parts of memory.

Re:Impossible

Nice link, I especially found the quote Can you conclude that "Gentoo is faster than Mandrake?" No. This is a limited test. It is likely that Mandrake is faster for some things. Also, we tested load-time performance only to be informative.

Re:Impossible

If you call a function many times, it will be in the cache most of the times that it is called. If you do not call it many times, any cache miss that might occur when it is called has negligible performance impact.

Inlining functions will increase code size because some functions will be inlined in many places. Increasing code size will generally increase cache misses simply because it occupies more space in the cache.

Still not any better than Debian

Compilation Enhancements

I hear the argument about faster executables a great deal, and it isn't a bad argument. However, Gentoo will allow you to install binary packages, and similarly Debian will allow you to install from source packages. More than that, you can actually find or create apt repositories with dependencies that are multiplexed across a set of architectures (for example, the nerim.net Mplayer repository; just tell it "mplayer-k7" and you get all the nice optimizations for Athlons with it).

Modularity

Again, Debian can be quite modular. Have you heard of Knoppix [knoppix.net] or Morphix [morphix.org]? They are very popular, and quite modular. There are probably more Debian derivitives than any other distro because they are so modular. I realize that Gentoo might also be good in this regard, but if it isn't provably better, I don't see a reason by Debian still wouldn't be a great choice for this project.

Re:why?

Ignoring for the moment that Debian is objectionable to no small number of us because of their explicit kowtowing to Stallman...

Gentoo is not just a source distribution. It is true that many folks treat it that way, and doing so has its advantages. However, if you don't want to compile everything from scratch to optimize it for your specific hardware, you can install precompiled binary packages and go to town. Look at the Gentoo Reference Platform (GRP) for details.

Re:why?

By caving in to his SCO-like demand to call the system "GNU/Linux", for starters. They also use his misdefinition of the term "freedom" as applied to software, and then there's their idiotic mishandling of KDE a few years back. They're more Stallmanite than any other distribution group out there.

You realize he's the one that invented the term "free software" right? I mean seriously, how can his be a misdefinition when he's the one that came up with it in the first place?

Or did you come up with a definition of free software and then pretend it's the real one?

Re:why?

Where this falls down is that true freedom must necessarily include the freedom to do things that piss others off, so long as it does not harm them. Stallman's version explicitly excludes the freedom to reuse code unless you buy in to his utopia. This is not freedom; it's a weak counterfeit. That is why I object to his redefinition of the term.

This is merely your redefinition of the term "free". You're touting hippie freedom, do whatever you want man, we wont stop you. Stallman touts speech freedom, do whatever you want with this code, but you can't stop anybody from doing the same thing to your code. His is closer to the traditional American idea of freedom, everybody has the same rights as anybody else and your freedom stops the second you try to infringe those rights. So again, yours is the redefinition. Yours has no concept of rights or limits on freedom, which is intrinsic to the traditional idea of consitutional freedom in America.

Re:why?

"Similarly, Stallman disallows some reuse of his so-called "free" code because he disagrees with it."

He does no such thing. It's a license. If you don't agree with it then don't distribute it. That's your freedom.

"Further, Stallman's idea of freedom is not truly free because it denies freedoms to programmers who wish to actually make money from the fruits of their labors."

Again you are completely wrong. The GPL does not prevent an author from making money from the fruits of their labors. It prevents them from making money off of MY labors. You see the difference?

"To the rest of us, it's a hollow shell."

Only if you are too stupid or lazy to write your own code. If you are leech on society who likes to kick back and let other people work hard and then suck away their property for your self then I can see how this would not work for you. Too bad America has become so used to getting welfare from the govt because now they want welfare from everybody else too.

Stallman or anybody does not owe you code. Nobody owes you code. You need to get off your ass and work instead of whining about how you are not allowed to make money off of other people's code.

"I believe the Founding Fathers would agree with my view on the subject, because they universally held that free speech must extend even to those concepts that some find offensive or objectionable."

Absolutely. Nobody argues with that. But apparently your limited IQ is not able to grasp a simple concept. Code is not speech. It's not protected by the constitution.

Finally ask yourself this. Why do you demand the stallman give you his code. Don't you think MS or Oracle have more and better code then stallman. How come you are not throwing a hissy fit because MS won't give you code? Aren't they infringing your right to make money?

Re:why?

Once again your tiny little feeble brain is confusing speech with code. Stallman and the FSF are very much likely to defend your speech no matter what you say. What they don't like is you stealing their code. Advocating freedom is different then advocating theft. No proponent of freedom says that you should be free to steal other peoples stuff.

They don't have the same definition of freedom that you do and you are really pissed because of that. So pissed off that you want to "disinfect" them. I suppose that means killing them or something because I don't know how else you would disinfect something.

Somehow you have gotten into your head that freedoms means you can do whatever the fuck you feel like whenever you feel like it. I hate to break this to you but that's not the way it works. Having absolute freedom means denying everybody else of their theirs. You are not free to rape women because you think they are pretty and you are not free to steal other peoples code. Just because you are not allowed to rape your next door naighbor that does not mean you are not free. No definition of freedom allows for something like that (except yours of course).

You should read the speech professor Moglens speech at harward. Their idea of freedom is one of a self healing commons. An enduring and irrevocable freedom. In order to achieve this kind of a perpetual freedom they have invented the GPL and you know what it works.

The reason you are pissed off is ample evidence that it works. You want to make money off of their backs. You want to take their code and make it your own, you want to sell it and make money off of it and they won't let you. By preventing leeches like you from stealing from the commons without giving back they ensure long term existance of the same commons.

Re:why?

Um, no...

By caving in to his SCO-like demand to call the system "GNU/Linux", for starters.

This is not a SCO-like demand. SCO hasn't shown any proof of its claims. RMS's claim, OTOH, that the current distros have far more GNU code than Linux kernel code, is a fact. Go look at any distro if you don't believe it. And stop trying to use bandwagon buzzwords and name calling to make your arguments. Actual facts work much more nicely. If you want to prove it, go do a GNU vs. Linux SLOC count (or whatever metric would make sense) of a major distro.

I metamoderate every Troll moderation I see as unfair for this reason.

Two wrongs don't make a right.

They also use his misdefinition of the term "freedom" as applied to software

[snip]
Disinfect the GNU General Public Virus!

Maybe I'm wrong, but you apparently just don't 'get it'. We live in a society increasingly dependent on technology. If our basic human rights become exclusively delivered by tech (they become more so daily), and tech is one big maze of toll roads, then we lose those rights. RMS doesn't misuse the word freedom; he apparently understands its meaning far more deeply than you do.

You are a troll, sir.

What makes you a troll is comparing Stallman to SCO. Sorry if you can't see that- it's not your beliefs that make you a troll, it's the shrill and foolish way in which you shoose to express those beliefs.

I'm sure you'll counter with "well, stallman's action in this case is like SCO." Well, if Stallman had his way, what SCO is doing would be not only illegal (which it probably already is) but vigorously prosecuted. Your comparison papers over differences that are as wide as the ocean.

Why Gentoo

The point of Gentoo is that using the source for installation allows much finer grained dependency resolution.

For example, take vim. Depending on what you have installed, it may or may not have Perl integration, Python integration, an X UI, ctags support, make or ANT integration, and so on.

A binary distribution needs to provide a different binary for every possible combination of those, if it's going to allow fine-grained choice around what the Linux system has installed. Either that, or you have to turn off a lot of functionality which could be turned on, in case the dependencies aren't installed.

With Gentoo, the binary's dependencies are determined at install time, so you can have a single package which supports all the possible combinations of other components the user might have chosen to install. If I have Perl but no Python dev tools and opted not to have Python integration, no problem, vim is built appropriately from the same package everyone else is using.

In practice, the binary distributions seem to provide only two versions of vim, a "minimal" terminal-only one, and an "everything, including X" version. Personally, I don't want either of those--I want most things, excluding Python and X. Gentoo lets me have that, Debian doesn't because it doesn't have a vim-perl-ant-make-nox-nopython package.

Re:Why Gentoo

Actually, Debian is close.

$ apt-cache search vim | grep vim
kvim - Vi IMproved - KDE 3.x version
vim - Vi IMproved - enhanced vi editor
vim-doc - Vi IMproved - Documentation files
vim-gnome - Vi IMproved - GNOME2 Version
vim-gtk - Vi IMproved - GTK2 Version
vim-latexsuite - Brings the LaTeX power to Vim
vim-lesstif - Vi IMproved - LessTif Version
vim-perl - Vi IMproved, with perl scripting support
vim-python - Vi IMproved, with python scripting support
vim-ruby - Vi IMproved, with ruby scripting support
vim-scripts - plugins for vim, adding bells and whistles
vim-tcl - Vi IMproved, with tcl scripting support
vim-vimoutliner - a script for building an outline editor on top of Vim
vimacs - Emacs emulation for Vim
vimpart - Vim Component for KDE

And you could always choose to compile the thing from source yourself. But I prefer the convenience.

(btw, I'm posting from a Gentoo machine.)

Re:Why Gentoo

If you're using gentoo, then you should know your USE flags are set in /etc/make.conf and they turn assorted features explicitly on or off. (Anything you do not explicitly specify will go to defaults.) Here's mine:

USE="crypt -cups curl doc flac imagemagick\
jpeg mbox mmx mpeg nocd pam perl png postgres\
python readline samba ssl tiff usb"

Note that as my USE flags are currently set, among other things which are defaults, vim will get support for python. If I put tcl, gtk, lesstiff and so on in my USE it would automatically be compiled with support for each of these things.

In other words, if you put an easier installer on the front of it, and slapped together a cute little program to help you maintain USE flags from the list of available flags, then you would have the very distribution we're talking about here. What's more, when you change your USE flags, as packages are upgraded they will be recompiled with the new flags, and support for new software. All you have to do to update software to support your new stuff is to re-emerge it with the new USE flags. You can rebuild the entire tree with emerge -e world, though I'm not sure what order it will happen in. You probably want to rebuild, say, glibc first with whatever your current gcc and binutils is, then the toolchain again with your new glibc. The complete system can be transferred by doing an emerge -eB which will create all the binary packages you need. (If compiling for a different architecture, you will need to tweak certain variables in make.conf on the command line using the 'env' command, or perhaps several 'env' commands, including changing the location of PKGDIR.)

Having a bunch of different premade builds of vim is one way to go, but with the current power level of even most portable devices a recompilation scheme begins to make a lot of sense, especially if you can do it in the background while doing other things, as you can for most upgrades.

I'm putting gentoo in a virtual machine for use as an application server for my Midori Linux-equipped (M4I actually) i-Opener, so that I will be able to ssh someplace and run some applications. I know that I will be able to optimize it to minimize its impact on my PC.

Re:Why Gentoo

Hm.

An interesting attempt would be to combine the source and binary packaging systems at the distribution level. You noted well that having all the different variations as binaries would require countless binaries to be distributed. The following solution would be a bit more involved.

Let's have a distribution, Distributimized Linux. In the package management system (or website or whatever), a user can click on a package they desire. This brings up a menu (or a screen or a page) in which the user can select the configuration options, dependencies, optional features and so on. Satisfied, the user will send in the request to receive this compiled to a binary.

This could be done directly by the distro computing farms but since it might be a bit too intensive for one party to handle the compilation of hundreds of packages daily, a better option would be to force the use of something like distcc for anyone using the distro. The central package management multiplexer would form the distributed compiling network from suitable computers and set it to work on the build. Then -in considerably less time than compiling it on your own- the binary would be dropped into the requester's computer, it would just execute the make install.

A problem to overcome is overloading -an individual computer should not be used in a distcc (or whatever) network more than X times per hour (could possibly be configurable at the user end (for example heavier loads when you're not there) but ensuring some minimum value) to ensure that any single system would not be bogged down. Another great advantage would be if each computer could build packages for at least one other platform, so that my x86 box could support compilation for Joe's KDE package for his Solaris.

I'd be happy to partake in a distribution like that, be it making one or using it :)

Re:Why Gentoo

>Your "Multiplexer" is analogous to a Bittorrent tracker, basically you are trying to apply a peer to peer model to dmake or distcc.

distcc, in essence, is peer-to-peer. I'm not proposing using distcc as-is, but it's an example of the solution, and it works well for its part. Much of the complexity (scheduling, networking) is contained within that application -the only thing the multiplexer needs to do is to maintain a database of available computers.

>Even if you were to wave magically produce such software and install it on ~10000k PCs, I still don't think it would be efficient enough to work.

Have you tried compiling an application using even ten computers instead of one? Even with such small grids the speed increase is dramatic. See the distcc stats for examples of compilation speedups. Certainly, large amounts of available computers would be a great help.

> Anything the user fetchs gets precompiled in the background when the CPU is idle and saved for later.

This assumes that all compilers (or, alternatively, operating systems) are completely rebuilt so that they can work in the background -this is not something compilers are known for. And this will take probably exponential time compared to the current situation.

>Sure, it will require some extra space to store binaries that have not been installed yet, but that is trivial, especially compared to the storage, bandwidth, and complexity issues of YOUR idea.

'Some' extra storage if you precompile all the packages of your distro? You're using Slackware 2.0, right? I do conceed that some bandwith is required in my idea, but I do not understand how the storage would be an issue?

>Um, your idea is much _MUCH_ worse in this regard, but I agree it is important, which is why the GUI I propose would allow you to set all these friggin settings on a per package basis.

Okay. This is my problem (and this is what my last comment went towards): how do you set the settings in these packages? You would manually have to A) select the package and B) change the settings, then C) schedule it for precompilation. We can already do this. The only difference is that the compilation will be much slower and that you'll have a fancy GUI.

Additionally, I don't understand why my idea is bad for options. Each build will, as I wrote, send the compilation options (say, USE variables and compiler settings in Gentoo speak) along with the translation unit so that the package is compiled just as the user wanted it to be.

>The whole POINT is to make it easy for the user to compile stuff with the settings they wish.

That may be your point. My point was enabling users to easily select the compilation options they want and make compiling from source a viable option for everyone. I certainly agree that a nice GUI is an awesome thing for a package management system, but it's just that -a facade. It's not going to actually change anything.

>Try coding this beast yourself.

I would be glad to partake in such an endeavour -I'm not a kernel hacker nor a distributed computing guru, but I'm sure I could help.

I can be reached at spam-slashdot [NEARBY] elvendesigns [PERIOD] com.

Re:Why Gentoo

First of all, a well designed application shouldn't have that kind of conditional compile time dependancies. It's possible to do that all at runtime.

Secondly, debian supports building from source right in the debian package system. It's not possible yet to build the entire distribution (due to incomplete and circular build dependancies), but when you've installed a base platform, it's quite easy to rebuild the stuff you need with whatever optimizations you prefer, all while still making it easy to do binary only installs.

Admitted, that last bit of functionality didn't really take off until gentoo led the way, but I remember compiling my own optimised debian packages in the previous century, so...

Um, what?

A binary distribution needs to provide a different binary for every possible combination of those, if it's going to allow fine-grained choice around what the Linux system has installed. Either that, or you have to turn off a lot of functionality which could be turned on, in case the dependencies aren't installed.

There are thousands of apache modules out there, but I can get pretty much any combination of them without recompiling. What you're talking about is a design flaw in vim, not a fundemental fact of computing. Look at Emacs with it's LISP based adons. No recompling needed.

notice the author is Ian as in Debian

This article is by Ian Murdock, who is the Ian in Debian [debian.org]. The logo isn't there because of a direct relationship to the subject of the article; the Debian logo is there because of a direct relationship to the author.

Notice that his current project (Progeny) is about companies looking to build on a 'distribution neutral platform [progeny.com]', and the link in the article goes to a page about 'Progeny Componentized Linux.' Believe or not Gentoo is not the only highly configurable linux game in town: Progeny seems to be playing that game, but at the enterprise not the consumer level. He's definitely not thinking of Gentoo for this role. He's talking about Progeny.

Kinda Cool

This seems like a pretty good idea, I am sure that many people would adopt it. However, it sounds an awful lot like Linux From Scratch, or Gentoo. I'm assuming that a distribution like the one proposed must be a binary one to appeal to the masses.Overall, though, this sounds like a good way to attract more people to the Linux community.

Uh..?

What I don't get is how this is different from, say, Debian or Gentoo at all. At the end of his blog he says "If this sounds a lot like Debian, that's because it is in many ways", goes on to list the ways, and then doesn't list any differences other than an Anaconda installer. So, is this debian that installs like redhat and lets you choose packages? I mean, it doesn't sound like there's anything new here at all.

Re:Uh..?

It does sound exactly like my web server, which runs on ancient hardware and a small drive just for the hell of it. Once I've done a debian base install I strip even more out, and then include ONLY the exact services I need, even then stripping out what I don't, while also rebuilding a kernel that has every option I'll never need removed. Sound? out. IDE support? gone. It reduced the kernel size, reduced the base install size, and boosted speed in serving pages (the job it does) by 10%.

The only difference in my mind is an easier way to do this componentizing than manually, package by package, but that's practically what Gentoo does already.

Re:Uh..?

AND

it makes your machine more secure because:

A) You have less services, so less chance one of them is going to be hacked.

B) You have less programs on your machine so less (I did not say none) chance somebody who DOES break into your machine will be able to do any actual damage.

Bryan

Re:Uh..?

It's different because, while Debian components work on a package level, he's talking about a modular level.

That is, a collection of packages that work together thematically. For example; a simple productivity module which includes mozilla firefox, evolution, and openoffice. Or a multimedia module which includes xmms, mplayer, and a smattering of DVD players. Or a server module, which includes apache, samba, et al.

It's like turning the course focus on a manual microscope instead of the fine focus. You get more faster, but it's not as accurate to your specific needs.

Re:Uh..?

It's different because, while Debian components work on a package level, he's talking about a modular level.

That is, a collection of packages that work together thematically. For example; a simple productivity module which includes mozilla firefox, evolution, and openoffice. Or a multimedia module which includes xmms, mplayer, and a smattering of DVD players. Or a server module, which includes apache, samba, et al.

Kinda like tasksel [debian.org] then?

Jay (=

The Rock Linux distribution build kit

Many options are selected at compile time, rather than in configuration files, for instance processor selection. My php configuration includes "--with-mcrypt --with-gd --with-jpeg-dir --with-png-dir --with-freetype-dir". The number of different downloads for any pre-compiled distribution will be enormous.

Rock Linux [rocklinux.org] isn't a Linux distribution: it's a distribution build kit, that allows you to build your own tailored distribution from sources, with your choice of configuration options.

Even if there aren't currently the options that you want, the simple text-mode configuration files allow you easily to add your own.

Are we talking about Morphix?

His idea sounds very close to Morphix [morphix.org]. It allows easy building of customized live-cd distributions. It supplies its own installer too.

Re:"distribution build kit"

Correct me if I'm wrong here, but from the Rock Linux manual, it looks like installs work pretty much like they do for command line installs with ANY source based distribution, just that the installer script includes a small extra section to copy all the stuff to an ISO.

The power of Rock isn't in installing a single package, built from source, on your system, though it can do that.

Rock allows you to create your own bootable CD from which you can install your own custom Linux distribution.

1) download Rock (mostly shell scripts and configuration files, and a *very* small number of patches to packages)

2) unpack it

3) select your configuration options - choose from a range of targets - minimal LAMP server, desktop, or create your own list of packages - select your target processor, and any configuration options you want - e.g. build postfix with mysql support.

Some of these are available as tick boxes in the curses based configuration tool, if not you can easily edit a text file.

4) download the sources you need

5) start build

6) drink beer, sleep, whatever

7) create ISO image, burn to CD

8) boot from CD, use curses based installation and configuration tool to install new system.

When you building a large number of boxes to be shipped to customers, and over which you want total control, Rock is superb. I can strip my distribution down to the bare minumum, and easily apply only those security patches or upgrades to new releases of packages I have tested.

No no you fool

One size fits all is good. That is why everyone knows how to use a PC (and Windows) and why Unix is a support nightmare. The last thing we need is another Unix/Linux dialect.

Ode to a Misshapen Bathrobe

One size fits all,
Be you short or be you tall,
Be you wide or be you slim,
Be you her or be you him.
Now please, don't start to scream and yell,
We never said it would fit well.

There are times and places where one size fits all may be vaguely suitable for a good many, even the majority of, people. If one happens to be exactly that "one size" you might even wonder why anyone would ever want something else.

There are also, however, times when one size fit's all, no matter how close the fit, is simply intolerable and a wee bit of tailoring is in order.

If you don't feel the need of another Linux "dialect" than ignore it. Those that do may find the new "dialect" finally makes life bearable.

KFG

Sounds like SuSE

They started by making products designed for single roles, with a database server, a groupware and messaging server, and a fileserver.

Modularity is great for large organizations, but at this point it would be foolish to fall into MS's line of thinking, that you need a separate server for each role in the industry. It would behoove us to try harder to break down the barriers between servers so that they can act in a cohesive, stable and seamless fashion, whether there is one server, five servers, or five thousand servers.

And that's why we need a stronger LVM!

Has this guy done any research?

To allow optimization for depth, a new kind of distribution is needed--a componentized distribution from which users may build platforms from the bottom up, including only the features and technologies their products require.

I don't know about the distro he's using, but my build of Gentoo [gentoo.org] only has the packages I want (plus their requirements).

The bottom up model is being used by distros other than Gentoo, too. He's not breaking any new ground or creating any unique ideas IMHO.

Re:Has this guy done any research?

umm.. something tells me, he probably uses debian ;)

So that's Gentoo without the compiling then...?

Sounds like a great idea - but surely that's what you get from the gentoo linux system - you custom build a verison of linux that not only has 'just the components you need' in it, but also is (or can be) specifically tailored to suit your hardware and peripherals etc. I can see an avenue for component based distributions taking off, however. The two challenges with Gentoo are 1) the need to compile everything from scratch (which can take ages) and 2) the almost vertical learning curve required to get the resulting linux system to work (work out of the box? - not really!). Presumably the component model might allow both of these to be addressed...

Distro for Users or for Publishers?

Interesting. I'd always felt that this is how Linux really works the best., rather than being a giant 1 gig hunk of software, I can pick and choose the parts I want to play with. This leads to lots of mistakes early on, but over time, you learn how to optimize and reevaluate what you need and where, with the end result of understanding your system that much better. So my question is: Was this a suggestion for Linux in general, or a suggestion for a new type of business model?

Re:Distro for Users or for Publishers?

I think this is the way security works best.

Let's imagine we have a monoculture of Linux boxes, all quite similar, all based on a huge install 'dump' of one massive base system.

There's a lot there for an intruder to play with. Makes it easier even, for automated intruders (worms etc) too.

Now, imagine there's Linux as a majority, but split into so many different specific tasks that there's few similarities between them, except for a micro base system; where even the kernels differ in their capabilities due to function. it CAN'T ever become a monoculture even if the same 'distro' group were preparing these systems, unless suddenly the entire world was running webservers, or were running desktop office machines, or running desktop home machines, or running as cluster nodes etc.

Of course, the extra fiddling around of several well-defined task based versions of linux is a pain in the butt then, but hey - just a thought.

Re:What's he on about?

The only appreciable difference he could offer (which he did not, but could) is binary packages instead of having to compile everything from source.

Of course, part of the reason Gentoo is from source, and why less modular distributions are so monolithic, is that many UNIX programs require specific options at compile time to modify their behavior to fit just right on your system.

Apart from having a huge compile farm which you'd hand the equivalent of your USE flags in Gentoo, and get back a binary built just right for your system, I don't see any particularly clever way to do this.

Good...but....

In order for Linux to appeal to the masses, these "choices" must be available in an easily installable "package". It would be great to install only those options you need, want or require. And, I think most important, is the time it takes to set up the system.

Morphix Plug

This is just what Morphix [morphix.org] allows you to do. It basically takes away the hard work of re-mastering a Knoppix CD.

Morphix is modular, and can be adapted with less effort

The base, the Knoppix part contains the kernel, kernel modules, hardware detection, etc. This base is left untouched. You can either a change a mainmod or add lots of minimodules.

The are four basic images [sourceforge.net] to start off with. So making you own LiveCD is much easier.

It even possible to save you files, configuration and setting to the Morphix CD you using, ready for next boot up.

Did I mention the GUI installer ...

Brendan Mentioned before and here

Re:Morphix Plug

I would like to see a really high quality LiveCD that installs a debian system smoothly.

Nowadays I use Knoppix to install Debian. Just boot up off the CD. Configure partitions, mount the partition, then run "debootstrap" specifying the distro you want and install location. Downloads the latest stuff from Debian and installs it.

That gives you a real Debian install and is the only way I've been able to install the unstable or testing Debian without upgrading an older install.

Relatively simple (assuming you know how to create and mount partitions) and works a hell of a lot better than the ass Debian installers.

Sounds rather cool to me.

He says a basical iso is avaible and "More components and a component-aware, Anaconda-based installation mechanism will be added in the coming weeks".

Heh, compiling everything for oneself through an intuitive gui sounds pretty cool to me !
Gentoo now accessible to the unwashed masses (which I'm part of), Yay !

Security

I think it would be good from a security stand point to be able to quickly build the most minimal system, but there is still probably a lot of stuff in the Kernel that isn't needed. Still it would be great to have a tool that was based on the reserve of package dependency and removed everything you didn't want/need.

configure before you download?

Wouldn't it be neat if you could go to a website, enter in a list of all the hardware on your computer, enter in the applications or types of applications you want to use, and then download a customized installation CD with only what you want included? Then if you changed any hardware or wanted more software, you'd revisit the site, enter in the changes, and then download a patch including required modules, applications, and a script that installed/configured the changes?

That would be cool.

Re:configure before you download?

Maybe Google could weild their might in this one?

They're big Linux users, so they obviously believe it's good for some things at least. They also have such a large userbase that this service could conceivably dent Windows' market share if advertised properly on the front page. They have such an enormous backend cluster that it should survive, and they could subsidise more servers by showing adverts during the download.

ROCK linux...

... is, in my opinion, one of the more interesting Linux distro's around right now.

Its not so much a distro, as a 'meta-build system', for building and packaging your own distro.

To me, this is the best solution, and while these sorts of build-system efforts are still in their infancy, I can see a day when you just answer a few questions, press a button, and get a custom CD designed -exclusively- for the application you've defined.

That's pretty nice. As a Linux user since the minix post, I'm excited about more and more of these sorts of 'smart build environments' becoming the 'distro construction set' de jour ...

My Idea

Would be a program that would generate a Linux distribution based on your desires.

It would be a wizard that would ask you questions about what you want. For example, do you want a server or a desktop distro, do you want KDE or Gnome, do you want office software, games, web browsers.

After you answer all the Questions it would make you give it a Name, Such as MooKore Linux, and it would genreate an ISO filled with the RPMs for you for you to install.

correct me if I'm wrong

But isn't/wasn't this what BeOS intended to do? On the one hand it would be nice, it would be compact as opposed to having 3! cd's full of stuff, yet at the same time, they'd better have a squadron full of developers who would change things on the fly considering the speed at which things change.

Hmm, sounds like

The (varying) approaches used by source distributions (Lunar [lunar-linux.org],Source Mage [sourcemage.org], orGentoo [gentoo.org]), with varying approaches, strengths and degrees of success?

Diversity is certainly a strength of Linux.

Good luck distro number 999.

While it may or may not be a good idea to have endless amounts of distros from a commercial perspective it is certainly fun and who knows maybe it will be just what is needed. Or not.

Point is it doesn't matter. He has an itch, didn't see anyone willing to scratch so is doing it himself. Maybe it will satisfy others peoples itches as well but if it doesn't it doesn't matter. His itch is being scratched. A non-commercial distro with 1 user who is satisfied is 100% succesful.

Better then all those whiners who want someone else to fix their problems.

But isn't redhat and mandrake and suse modular anyway? Not like they force me to install apache or a window manager. Just the if I want say xmms I bloody well going to have to install X for reasons that should be obvious. You may want MS to stop bundling IE but then don't go complaining that Windows Light doesn't come with a browser installed.

As for putting everything on the CD. Well yes I thought that was pretty nice. Since they want you to buy the thing it means that people with modems don't have to download several gig of extra data just to get a working desktop. KDE is about the only real offender insisting on installing games on every distro I tried.

Re:Please sir, can I have some more?

What you're arguing is that for Linux to succeed, it needs to become something other than what it is. The whole point of open-source/free software is that there is no One True Package.

If by some Act of God, Linux were to become what you say it should, then the people who currently use and support it would instead switch to BSD or Darwin or OpenBeOS or some other open, polymorphic OS. Because that's what they want, not Linux for the sake of Linux.

The down side of freedom is that you need to make choices, and the down side of making choices is that you need to do some research. If you want a developer to do all that for you, then you should stick with an OS from company that will do that. Microsoft excels at it, and if that's a little bit too controlling for you, you should be quite happy with Mac. (Seriously, OS X is a great centrally-designed system.) But what see as "what's wrong" is what others see as "what's right". Deal.

Gentoo is not the answer.

Gentoo is compartamentalized, but not in anyway that other distros such as debian isn't.

Both Debian and Gentoo are heavily optimizable, you choose the componates that you want etc etc.

Bot have your advantages and disavantages.

Debian's is that the developement cycles that forever to make sure that everything is working correctly, but you get a reliable computer that is usefull for hundreds of different applications.

and Gentoo's big disadvatage is that it's worthless for anything other then home desktop, but you can play around with newest technology.

(could imagine administrating a hundred gentoo boxes buy yourself and getting someone to actually think it's a good idea to pay you to run a OS on them that takes a average of two days to get installed?)

And no compiling for speed is DEFINATELY NOT WORTH IT JUST FOR A PERFORMANCE ADVANTAGE in 95% of the apps you would use on a daily basis.

But IMHO people are naturally moving towards comparmentalized OSes anyways in Linux. Weither or not they realise it.

Think about, APT and other decent package managers have caught on in a big way. Fedora can use both Apt and or Yum.

Using package managers it's easy to customize any install and the BIGGEST advantage is that it's simple to keep everything up to date and to install new programs.

A BIG advantage over closed source stuff. (once you get it set up.)

Now if most linux distros agree to stick to a common Filesystem Hierarchy system (http://www.pathname.com/fhs/) then you can use all sorts of packages together.

I could use Fedora packages, apt packages, debian packages, gentoo build scripts and all sorts of stuff and pluss get support for closed sourced software easily in any distro of my choosing.

If Debian doesn't have a new enough XFree86 build you can install it from Fedora and build the latest KDE 3.x beta from portage scripts from Gentoo.

That's what we should aim for, and a common FHS is pretty close. People are beginning to learn the best way to do stuff and the directory systems are beginning to be more and more common to all Linux distros.

In a few years I hope the consept of numbered linux distro releases will be gone and we will move to a stable/unstable model similar to Debian.

Re:Gentoo is not the answer.

I maintain 3 Gentoo servers (1 of them LTSP, the other two file/print/web) One of the servers has 4 gentoo desktop machines connected to it. I have a portage/distfiles NFS share, I use distcc and I find it really, really low maintenance. My desktops have kde 3.2, publishing stuff, OO, while the servers have Apache, Qmail, etc. Very different installs, all managed the same way. After 4-5 years of Redhat the /etc folder was still very mysterious to me. After 4-5 months of Gentoo everything began to make sense. I switched to Gentoo some time around Redhat 8.0 (can't remember exactly, but my machines were running 7.x when I swapped 'em). Never had a problem.

Mandrake has tools for such

There is mkcd [mandrakesoft.com], which allows you to create custom Mandrake CDs with the software and options you want.

And mandrake has a customizable auto bootup/install via drakx (mdk's installer system).

Add all of the above, and a little knowledge about SRPMS (if you want true customization), and it works rather well. Also Mandrake's public download edition is 100% FLOSS, so there are no issues about redistributing the software (unless *you* add some non-FLOSS stuff on your own, heh)

Sunny Dubey

modularized distros

I think PLD [pld-linux.org] (Or in English [pld.org.pl]) tries to be highly modularized:

no restrictions for a set of packages that must be included in the distribution. The user can have access to every package already prepared for PLD. If something had been prepared in conjunction with other packages, it means somebody did need it, and maybe someone will need that package in the future

Now, this is not to suggest that PLD does this well, or that it does this actually implements what Progeny is suggesting, but it's still a starting point.

Linux and the fight for world domination

Although I dont think Ian raises any particularly unique arguments, the article is a susinct introduction to the elements that emphises Linux's strong points.

The thing that aroused my interest in Linux was not its cost, but its ability to be used in projects that were not limited to traditional PC software.

Imbedded linux will (as long as MS doesnt rethink its licensing) rule the non-pc computing world.

It makes perfect sence. Who cares how your C64 watch works, as long as it does.

It seems unlikely that "componentized Linux" is the answer because only imbedded linux realy needs to get down to the "Linux from scratch" kind of level - otherwise, you'll probably be looking for a higher level distro.

What does he mean?

Wow - he managed to write 600 words without really explaining what he wants to do.
Is he just saying that distributions should go for niche markets by allowing greater customisation? So instead of installing everything of the 3 CDs you only install what you want? Kind of like every other distro?
Or is it more than that? Some kind of pluggable component system akin to Debian's virtual package "provides" system? So you can have different packages that provide standard services (mail, desktop, web-serving etc.) through common interfaces to the other components.

Folks, stop the fanboy stuff

People keep saying "should be done like Gentoo" or "Debian is like this".What Ian is trying to say is everyone needs to cooperate on this and build a framework which all distros can use.

In my eyes one of the problems Linux has is libraries and their versions. you can't simply take an executable and guarantee it will run on another Linux installation (unless you statically link).

We have this already.

It's called UnitedLinux

Blinding flash of light!

God! Yes! That's it! Why didn't I think of it before? That's what Linux and Linux users need... Another distribution.

Sounds like (insert denomination here)

So far I've seen a lot of posts where people are already saying Gentoo, Debian, LFS, etc...

Almost all Linux distros are componentised. OK everyone let's hear it: "Linux is not Windows."

We've got distros mainly because we aren't all kernal coders who know all the in's and out's of every single chipset. Quite frankly I don't know who even has the time (but apparently some of you do). We have generic groups of packages/aptget/emerge/etc. to allow for faster deployment. And that's another beautiful part to Linux: Choices!

Yes, perhaps it's overwhelming at first, but you can build it from the ground up if that's what you really want or just pile it on thick and zesty!

The author wants to promote Progeny and "Componentized Linux", and I think there's always room for Yet Another Distro (YAD), but to say the others are doomed to fail because they came on 3 CD's (Think Fidora) is misleading. Mandrake 9.1 came on 3 CD's but it certainly won't force you to install all of it. In fact, you can just select a kernal only option, and it won't even ask for the other two disks. Not only that, but you can hand select only the packages you want. How cool is that?

So I guess what I'm trying to say is that most linux distros have options to allow their users to build it pretty much from the ground up. The reason for the different distros lies in what their vision of the ultimate system looks like when it's totally loaded down.

Big Whoop

Can somebody who actually READ THE ARTICLE tell me how much more moduler than apt-get install packageX it can be?

Sounds to me like a front end "Install Web Server?" "Install Development TOols?" choices that proxy a few packages is all this is about.

Aren't all Linux distros these days already got some sort of package managing that manages every file? Even the base Libc? How more moduler can you get???

Marketing Hype???

Isn't this really more along teh lines of marketing hype then it is general user useful?

Come on now, we no longer have sub 50Mhz CPUs (but many times that and getting faster),all the expensive backup media we used to use has been replaced with CD/DVD writers (and that's only going to improve), general storage/access media (hard drives) are far more massive in storage space than the old 120Meg drives and and even far more inexpensive (the larger the drive the cheaper per Meg you pay)....ETC...

And HEY, we can even use more ram bits for dates, avoiding things like Y2K... Or is not gigs of ram not enough?

Is it really a value to have injected additional parts complexity to have to deal with?

What is the trade off? You use up a little less drive space, maybe make a fraction better use of your CPU, use a little less ram space and backup media....in exchange for....

Additional complexity to allow you to screw things up more often...

Hell, just wait 6-18 months and get a new faster, larger storage, more ranm, etc... system.... The cost difference will be less than what you might spend in maintaining componentalized linux.

Hell, I really like the Live CD concept, where it determines what hardware you have and auto-configures.....but all from a standard full package.

I'd like to see the opposite

The problem we have here is that linux is designed for linux users. Like myself, I prefer gentoo. It fits my person style and I just love emerge-ing all kinds of junk and making my own kernel.

I would like to see a linux distribution the exact opposite. One that I could give to people fed up with windows. It should detect all the hardware like knoppix. Then it will bring up a simple GUI style disk formatting tool, like the mandrake installer. Then after I select which partitions it should just install, no more questions asked. When its done all the hardware should be working. One of every necessary software application should be installed. The gui will be simply laid out with big pretty buttons. One that says Web Browser, another for Word Processor, etc. Wine, lilo and other things will be configured perfectly and automatically without user input. There will also be another big button that says "install software". It will have a big nice easy to use app that sorts softwares by categories, shows screenshots and readable descriptions of different programs. With a single click these programs will be installed and new icons will be created. With another click these programs should also be automatically updated to the newer versions without breaking anything. And of course easy uninstallation is a must too.

I see no reason why this isn't possible. Why hasn't anyone (that I know of) done it yet?

Re:I'd like to see the opposite

SuSE Linux is mostly what you describe.

But indeed this is what is required for a desktop Linux.
No toolkit of modules, but a standard install that sets up a standard Linux installation that can be made user-friendly, can be well debugged, can be optimized w.r.t. parameter settings, etc.

Dear god.

There are thousands of fanboys waking up (or getting ready for bed :P) across America right now that are reading this, and all of them are thinking, "Well, this or that distro already does it!" You've all missed the point.

Has it occured to you that his writing isn't directed towards those of us that already use Linux? Could it be that the founder of Debian would possibly want to make a little money on his toils and ventures by selling his ideas to Suits and PHBs?

No, that couldn't be. Could it?

Yes. (And no, I'm not saying this is a bad thing.)

Stop thinking the world revolves around you (us) and your (our) zealotrous love of your distro. (Particularly you gentoovian freaks with your distcc clusters! :P) Seriously, though. Linux is linux; let's not make a fuss. It's just nice to see a movement away from the techniques of the past - RPM, in particular, which doesn't make custom rollouts terribly easy.

It's been my long-time argument

That Linux distros are taking the shotgun marketing approach, unlike Microsoft who has painstakingly researched what end users want in an Operating System and for the most part, has delivered exactly what the majority of PC users want. Granted, Linux is destined for the server market for the time being so a distro packed with services is appropriate for the most part, but if Linux ever wants any substantial share of the desktop commodity its going to need to do some serious work on several fronts like UI, ease of use, intuitiveness, size and speed.

Anyone use OpenNA Linux?

My organization is standardizing on it for critical servers, and I think it does a lot of what this article talks about. On install, it asks which services you want to run ... and it ONLY installs what is absolutely necessary to run them. It's pretty lightweight, but gets the job done. And it's also hardened like you wouldn't believe, with most services preconfigured to run in a chroot() jail, something the others should have been doing from the start IMHO.

Website [openna.com]

Why this can't work

There are several reasons that distros are built top down and you would think that Ian would know.

Linux packaging isn't bad at all, it is actually the lack of any standards that hurts the natural evolution of a modular Linux.

GCC/glibc are moving targets. You can't depend on linking between two versions of GCC or glibc, so all the apps we package today will be of questionable use tomorrow.

All other libraries suffer from the same problem. There is no guarantee that you can upgrade or install anything on the system without breaking random other applications.

There are far too many compile time options in applications. Instead of checking for dependencies at runtime and acting on that information, the applications have to be built either for a minimal system configuration, possibly dropping features, or built with every possible dependency, making installation require far too many dependencies.

Until these issues are cleared up, there is no other way to create a distribution than top down so that all dependencies are known and accounted for or built from source.

This has been one of my major gripes with Linux

It's monolithic nature has been one of my major gripes with Linux. This is most apparent in the kernel itself; the sources are distributed in one big (and I mean BIG) tarball containing sources for nigh on every architecture and every device supported. Then when you configure the beast, many options cannot be built as modules, so it's either bloat your kernel or miss out.

The same is true for many distributions. Although a lot of software comes in packages, installations tend to range from quite heavy to almost ridiculous (about 1 GB). And the kernel, again, tends to be a fairly monolithic one, supporting a few filesystems that are unlikely to all be used, etc.

I have to say that Debian tends to be quite OK. The base install is, what? 100 MB? And to that you can just add what you need, dependencies solved for you and all. The kernels you apt-get are usually modular (although the generated ramdisks haven't always worked for me, and cannot be edited due to their being in cramfs). Still, it's annoying that when I want a feature added to my kernel, I have to reconfigure, recompile (I don't' keep the object filesaround - they take too much space), reinstall, and reboot. Sure, I could get a faster computer and a bigger hard drive, but even then, having plenty of something is no excuse to waste it.

Bottom up? Isnt that debian?

I guess we should stick to improving debian as the defacto standard of Linux instead of building competition for it. Distros like KNOPPIX have shown how a good lowlevel distro can be used for higher level distros, keeping the base (debian) same allows for much more compatibility (dpkg).

Now if only SuSE and Redhat pays attention.

Re:hmm

How about more modular software instead of just focusing on the OS?

Everyone's going on and on about making the OS more modular. The problem isn't really the modularity of the OS. The issue that comes up is how *unmodular* the software is and that's why a from-source distro fits this scenario well. For a binary distro a lot of software needs to compiled for many different scenarios (such as the vim example for X, gtk, CLI, etc), or in possible cases compiled for all scenarios. For a from source distro, you only need to compile it to fit exactly what you need.

I agree that the OS needs to be more modular, and I agree that binaries need to exist for those evironments where it is necessary or demanded. However, I think that at the same time we need to consider writing our software to be more modular. If you didn't have to recompile binaries to support different scenarios, this modularity of the OS wouldn't be quite as necessary. If you could have one compiled core for vim, then attach different modules to it, it wouldn't be necessary to package the core 25 times.