Why Open Source Doesn't Interoperate

bergie writes "There is an interesting article on Advogato on why it is so difficult for Open Source projects to interoperate or support common standards. Often cultural differences between projects, egoes, and many other issues stand in the way. The article outlines some practical ways for improving the situation, based on experiences from OSCOM efforts to get support WebDAV, SlideML and other standards into Open Source CMSs. Examples of successful interop projects include freedesktop.org, the cooperative effort between GNOME and KDE."

Great Article on Open Source

Here [eightballmagazine.com] Interesting reading

Re:Great Article on Open Source

From the eightballmagazine.com article:

Of course they still need to learn to design with the user in mind. Make the software easy to use and install. The shorter the learning curve the better the design. And for gods sake write things that are what the end user wants... not what you want them to have. There is nothing more irritating to me then an arrogant nerd who can't understand why nobody will listen to him.

Maybe I've just missed the point of open source stuff over the last few years, but I always thought the idea was that people contribute to an OS project because it scratches their itch. For example, I find an open source database I would like to use in a project, but I need to tweak on it a bit to make it work on my platform, so I make the tweaks and contribute them to the commons.

I don't care if Joe Sixpack can't use the database, but then again I'm not going to whine because nobody but 'geeks' uses it. I know there are people that whine because nobody uses their hard-to-understand project, and they need to either stop whining or spend some time making their stuff more usable.

Companies worry about market penetration and product name recognition - and making money. When I work on a project, I usually only worry about the first two if I'm also worried about the last one. And most of the time, I'm just learning something cool, not trying to displace some commercial product. I don't like it when people like Uncle Pru get on their high horse and tell me that my goal should be to 'write things that are what the end user wants.'

Ooooh, I know the answer!

RMS is the reason! I GNU it all along!

Advogato's number

No, it's because there are 6.02 x 10^23 developers working on the system at any one time.

Because free software is not planned

To make software interoperate, developers need to create interfaces before writing their software. Often no planning is done and developers start writing their code without a clear vision of what they want to write.

There is a certain overhead in creating interfaces. They can take time to develop and they aren't any good, you'll be stuck with them for years. Even Linus is against the creation of standard interfaces internal to the Linux kernel. That decision inhibits the creation of a truly modular system.

Re:Because free software is not planned

Unfortunatly a lot of code writing is done with inadequate planning, but this is an inherant problem. Without a crystal ball you cannot predict every twist and turn you will face.
This is not just a problem with free, Free or open source software, but also with planned, structured development of commerical model software such as Windows.
After all if we could predict the future why would ever need new versions? :)
However I don't think that a neatly integrated environment is impossible, just difficult.
Besides, getting around integration issues is part of the fun!

Re:Because free software is not planned

This is not just a problem with free, Free or open source software, but also with planned, structured development of commerical model software such as Windows.

However, I would venture that there are more commercial products with some sort of functional and or design specification.

Not to say that these specifications mean anything when they're done by incompitent people.

I guess my only point is that it certainly wouldn't hurt to have not only more communication among OS development teams, but a little more planning about their interfaces. Why not solicit some software engineers who have background doing this sort of thing, and not just soliciting coders?

And lastly...Everyone seems to be overlooking the fact that so many people doing OS projects, do have the "not invented here" mentality. However, using middleware takes time to research and test, not to mention that you've got to have a better idea of how things interop before that's helpful. So, it does come back to the issue of planning.

Just my thoughts...

Re:Because free software is not planned

So you're telling me that all those managers have actually been useful over the years?

some sort of functional and or design specification
That just about covers some of the best stuff I've been given to work with :)

Re:Because free software is not planned

Often no planning is done and developers start writing their code without a clear vision of what they want to write.

...and this is different from commercial development, how? Or rather, is your company taking resumes?

Re:Because free software is not planned

As a follow-up to this, remember that the *vast* majority of software being developed is not commercial software. It is in-house proprietary software. And the planning that goes into a lot of it is dubious at best.

In most corporate settings it's like this:

PHB: "We need software X to do Y by tomorrow."
Developer: "But we need time to plan and design and..."
PHB: "That's why I said we need it tomorrow instead of today. Now get to work..."

Geez, don't you people read Dilbert?!

Oh, and BTW, what's an EGOE?

Re:Because free software is not planned

Your subject line says it all. Well, nearly...

I'd say the two biggest sins of the open-sourcers are
a) over-generalisation (it'll be able to do everything)
and
b) over-specialisation (it does one task, but can't do similar ones)

I'm finding it hard to think of examples, but I guess GNU grep's an OK example of something that's just about right.
Expanded to do enough things like context greps (e.g. give me 4 lines before the line containing "Name:" and 1 line after), and a few other features (e.g. '-c' so that you don't need to '|wc-l') that add to its functionality, so it isn't over-specialised. Likewise, it's not sed, awk or perl, they realised that just keeping it simple and lightweight was the way to maximise its usefulness.

YAW.

Re:Because free software is not planned

From an interview with Rob Short [mattschwartz.net], the vice-president of Windows Core Technology, regarding Microsoft® Windows(TM) Server 2003:

Why is there no command line only version?

We're looking longer term to see what can be done, looking at the layers and what's available at each layer and how do we make it much closer to the thing the Linux guys have -- having only the pieces you want running. That's something Linux has that's ahead of us, but we're looking at it. We will have a command line-only version, but whether it'll have all the features in is another matter... It's a very tangled subsystem.

<plug style="shameless">
As I also explain at MS Versus [mattschwartz.net], Bill Gates has testified in federal court that Microsoft® can't modularize their operating system or document all of its APIs because it's written by groups of developers haphazardly binding software together without any clear overall design.
</plug>

Re:Because free software is not planned

Well, I dunno about that. I've worked on any number of "planned" projects, and in every one, the same thing has happened: The managment comes along and changes the requirements. So you have to kludge the plan to handle the new requirements. Then they do it again, and again ...

I don't think I've ever seen a planned, commercial project that interoperated with anything else. Usually you even have to write new editors because all the file formats are proprietary. Just looking at the data in a file often requires a new program, because it's a new, proprietary format, and not even your vendor's software can read it.

OTOH, I have some "C/unix" programs that I wrote 20 years ago. I've reused them on dozens of different kinds of computers, and usually about the only problems are that the #include lines need to be rewritten. Sometimes I have to change a few variables to a different size int. But there is rarely any problem with my programs interoperating with other programs. This is shown by the fact that I always have a lot of scripts that combine my programs with other programs.

The project I'm working on now consists essentially of "raiding" the data at a big corporation, and extracting as much info as I can from their old, poorly-documented files that are in formats that can't be read on any new computer. My code converts them to plain text, mostly flat files, a few HTML and XML files. They can be read on any computer that we copy them to. They can be dropped into a web directory and read from any browser on any computer.

I'm doing all my work on linux boxes. I keep pointing out to them that my converted files no only work ("interoperate") with most any linux software, but will work on nearly any computer out there, and will continue to work for years. They believe me, because when they copy the files to their other random computers, their programmers don't seem to have problems with the file formats.

Of course, one of the things they really want is to take the data and stuff it into a commercial database. The result will be data that can only be used by a timy amount of software from that database supplier. Well, that and the perl DB package (which we will note is Open Source). But note that they are doing what corporate management always wants to do: Put the data into a format that can't be easly read by other programs.

In summary, I'd say that the Open Source environment, and more generally the non-proprietary parts of commercial unix systems, are by a large margin the best place to find interoperability. Closed, commercial systems can generally be summarized as "Your data won't work anywhere else, and won't even work on the same vendor's systems 5 years from now."

At least that's the history that I've seen so far.

Re:Because free software is not planned

Even Linus is against the creation of standard interfaces internal to the Linux kernel.

First of all it is important to keep in mind, that there are two different kinds of interfaces:

1. Internal interfaces between parts of the kernel and kernel modules.
2. External interfaces used by user mode applications.

The later are aiming against complience with UNIX and POSIX standards. It is only the internal interfaces that can be changed without notice. But notice that as long as the interfaces are changed thorugh the entire kernel users are not affected. It does mean you cannot mix code from different kernel versions, but end users are not supposed to do that.

Of course the changing interfaces are a problem for anybody who wants to develop closed source driver modules. But Linus and other kernel developers does not like closed source modules, the modules impose a problem on anybody who wants to debug kernel problems. So Linus certainly doesn't want to stop kernel development in some areas just to help developers of such closed source modules.

Interfaces are not changed without a reason. Some developers of closed source modules might think interfaces are changed to annoy them, but of course that is not the case. However they do have to understand, that support from the Linux community cannot be expected before they open their own source. Hardware vendors must understand, that writing a closed source module does not make their hardware supported by Linux, a open source driver is needed before full support is existent.

They say.. they say.. they say..

They say open source is good... They say open source is bad... They say open source is in-expensive when... They say open source is expensive when... They say open source is good when it comes to... They say open source is bad when it comes to...

I wonder who of them is actually using open source for anything than writing redundant articles about it... I can pull as many pro- and contras out of my ass when it comes to windows. But nobody would care. Nobody cares about because everything already has been said. I don't care. I don't read articles before I comment.

One factor is obviously...

...that open source authors prefer solutions they like over "standard" solutions.

Industry standards, particularly those created by committees, are often abominations that people only use if they have to. In my experience, the extent to which people like things like CORBA and XML often seems to be inversely proportional to their level of technical sophistication.

RFCs have much more respect in open source circles than committee-created standards.

Re:One factor is obviously...

heheh..

this reminds of a very nice company that has had an habit of making closed-source, lock-in products that mostly don't care piss about standards, rather just work enough to them to allow using other products but then be so broken/intentionally closed that those other products can't be well used with it..

open sourced can be just as broken as closed source, but you sure as f* are more probable to get your own product to work with open source solution than closed source one.

GNU / Linux

<sarcasm>
Yeah - let's just hope and GNU and Linux can get their act together sometime soon
</sarcasm>

It's the ego.

The hardest thing to break is the ego barrier. And it is very evident in many aspects in open source. Take for example the kernel source and different programs branching off. It's great, and even arguably beneficial, except for the fact that different versions of the same program are competing against each other where the forces should be united. Again this is the beauty of Open Source, where you don't have to rely on a leader/Dictator how ever you see it. The fact of this multiplied by the ego's in different countries just make this a big number. What also does not help is the fact that many programmers are not being paid such good money for the programs either. I will still use open source software at home and love it. If people want to compare M$ "standards" to Linux just remember one thing, M$ has about a ten year head start. ;-)

Hrm

You'd think if geeks couldn't "interoperate" with girls, at least they would be able to interoperate with other geeks on their projects.

*ducks*

OpenSource creates de-facto standards

One of the reason is that by releasing your source code you can easily create de-facto standards. VNC is a good example. It is not standardized at all, but the original implementation came with source code under GPL. Dozens of VNC projects have been created since, all inter-operable, and most of them are based on the original source code.

Re:OpenSource creates de-facto standards

Actually, there's a pretty good specification document for VNC. I was able to write a working implementation of the protocol (for keystone) that was not derived from the ORL sources. The documentation can be found here:
http://www.uk.research.att.com/vnc/protocol .html

The real reason it doesn't happen is ...

The real reason stuff doesn't interoperate is the people that want it to, didn't do so.

If I want my project to interoperate, I will incorporate that feature. If I don't care, I won't.

If someone else wants it to do something else THEY have to add that feature.
Why should I spend my time working on features for you?

This is free software, the interoperability people could make it work, but they don't put the effort behind it, so it doesn't happen. If it isn't worth their time, money and effort, why should it be worth mine?

Various comments

I have been involved a little bit with the OSCOM efforts and I am impressed again and again on how they all work together. The board members of this organization are leaders from various OS Web Application servers, all having different interests and yet they can work together.

I only know Paul Everitt (one of the authors) personally, who is co-founder and used to be the CEO of Digital Creations (today Zope Corp.) - therefore one of the inventors of Zope (www.zope.org). He started the Twingle project a while ago, trying to generalize the Zope effort to create a content management Mozilla-GUI for Zope 3 to all Open Source CMS solutions.

As the article states this effort is quiet ambitious, but it also shows the power OS can have. When Paul and I started working on the original code, we used heavily XML-RPC (it is just the easiest to use for getting anything done), but Paul has since pushed towards HTTP standards, such as WebDAV. While this is much harder (i.e. I am writing a WebDAV library in JS for Mozilla) than the original approach, it allows a lot of integration possibilities later. For example, in the future we imagine that we will be able to drag and drop objects between Bitflux and Zope and vice versa for example. Also, a unified GUI will allow Content Designers to gain a skill that is much less platform-specific (in the meaning of App Server and Operating System), which makes this skill much more marketable.

BTW, OSCOM 3 will be held at Harvard University on May 15, 2003 if I remember right. So everyone interested in Web-related technologies living in Boston should drop by and check it out.

Dubious conclusions using a limited data set

To be honest, the headline is sensational and the document itself has limited content from which to draw conclusions. Most critically of all, however is the assertion that Open Source projects refuse to interoperate based upon experiences of a monolithic organisation trying to get OSS CMSs to implement WebDAV.

To be honest, I've written several CMSs, contributed to others, and done 90% code re-writes on others to suit my needs. All OSS.

The thing is, you'll find that many Open Source CMSs don't always support LDAP or a host of other standards. Why? Because they don't need to. PHP Nuke, for example, is a fine project for producing small-to-medium community/corporate content-driven web sites. It isn't perfect, but it is modular, and a bit of work will allow you to produce some very nice, functional projects. It doesn't need to have to support another protocol, WebDAV throughout and then SlideML on top.

PHP Nuke is, in fact, a combination of other projects brought together and welded into a final package. That's interoperation for you...

What about the OASIS initiative, where open source projects are trying to produce XML-based standards for office documents to ensure long-term data access and inter-operability. What about X? VPNs, secure communications, file sharing, standard web protocols and a hundred other examples of OSS collaboration, where proprietory companies are digging their heels in to try and jealously guard their marketshare?

If you want to know why OSS CMSs don't have WebDAV support, it's because they don't need it, plain and simple. If it was important and really would make an incredible difference, they would all be supporting it. As it is, from what I can see, what is on offer is something that their code already does for the most part. They don't see the point of it, neither do their users. Oh, and they probably don't like you writing articles, saying that they don't play nicely when you arbitrarily come up with things and tell them they should implement them. :)

To everyone else, sorry for the rant, but this article really is nonsense and insulting to everyone who works hard in the open source community on almost any project.

And proprietary software interoperates better?

Can someone point me to the article that explains why proprietary, Imprisoned Source Software (ISS) does not interoperate well? Oh yeah, it's because of profit motives, monopolies, cultural differences, egoes, and "many other issues".

A vicious cycle is...

...the fact that doco is often nonexistent or poor, code is idiosyncratically designed/written and poorly commented (if commented at all).

Result is that quite often, it takes less time to implement something oneself than to understand and integrate with a 3rd party piece of software providing the same functionality.

Too many developers think it's beneath them to write good doco, example progs, tutorials, clear easily-learned APIs and clear meaningful comments in the code.

It's a kind of elitist 'techno macho' attitude - 'if it was hard to write, it should be hard to understand'. Too often my questions to authors of unfamiliar software are met with a terse 'RTFS!' (read the fucking source).

This syndrome creates a fragmentation, which destroys opportunity for leverage from well-collated and well-catalogued sharable components. Which in turn makes developers' time more scarce, and further discourages the efforts to make code approachable.

Blindered developers

One of the biggest problems I see with Free Software development is the problem of the blindered developer.

This is the guy who doesn't bother to raise his head from the computer to look at how his project works in any environment other than *his* system. You know, the guy who requires you to have libfoo.so.5.1.2.pl6-thursday-0741am-fred-mutant1 installed just to compile his code, and by $deity no other version of the library will work.

A concrete example: The developer of the GATOS project [sourceforge.net] (a driver for the TV tuner/video capture (but not video out) functions of ATI All-in-Wonder cards) requires you to use HIS kernel module and HIS radeon driver. As a result, you may EITHER use his code XOR the DRI accelerated 3d code, but not both.

True, he does (to an extent) track the DRI development, but rather than working with DRI and XFree and coming up with a way his drivers can play nice with the standard builds (e.g. having hooks in the standard driver and having the standard driver load his modules if present) he is off on his own little branch.

He also uses libraries and packages that are not part of the standard installs of common distros - as a result just getting his code working is a real slog. So many people don't do it, and his project does not get as much support as it might.

Now, I am not picking on him - developing stuff like that is hard, since it is very poorly documented. And with DRI making changes, XFree making changes, and him making changes, you WILL have times when things don't play well together. But rather than that being a transient state of affairs it is the normal state the GATOS project w.r.t. DRI.

Unfortunately, it take time and work to stop, get a fresh install of RedHat/SUSE/Gentoo/... and see what it takes to get your code to build and install. It takes work to make sure that you really NEED the latest version of libfoo, rather than just any version. Especially when your code interoperates tightly with other people's projects it is difficult to plan interfaces that won't change frequently. If you can accept help from others this isn't so bad, but many project "leaders" have the attitude of "HOW DARE YOU IMPUNE MY PROJECT! IT IS PERFECT UNTO ITSELF! I CANNOT HELP IT IF YOU ARE NOT 31337 ENOUGH TO HAVE THE LATEST STUFF! L@M3Rz! IT IS UNDER DEVELOPMENT!"

But that is the difference between a hack and a software engineer - just "getting something to work" and "getting something to work well, under as many circumstances as possible, as smoothly as possible."

Re:Blindered developers

One of the biggest problems I see with Free Software development is the problem of the blindered developer.

... but then one of the solutions I see with Free Software development is the solution of the enlightened (no pun intended) software engineer. For you can take this hacker's code and make it work in your environment, or backport it to DRI and/or XFree, and he'll probably be ok with it, as long as he doesn't have to support that

Is it me?

Am I the only one around here that really doesn't interoperate that much? Aside from pasting the goatsex man picture in my emails, I really don't interoperate that much.

I have been known from time to time to use two different text editors on the same file! Egads. ...
But I never do something like... Write a SQL Query, and want in to be executed from a Word document each time the user opens the file.

In fact, I rarely, if ever, insert a Spreadsheet into a Word document. I know I can. I just don't.

We're starting to use XML, SOAP, WebServices. But really don't need to. Why create then turn a Java object into an XML document, send it over the wire, then convert the XML document back into a Java object. "Well... It's because later, something besides Java will be interacting with the system." I don't think so. Everything new we're writing is in Java. So, just regular Java RMI (EJB) will do.

Because it's hard?

I'm trying to create a packaging metadata spec at the moment to allow standardised LSB RPMs to be installed on all compliant distros using native dep resolvers etc - in short, making a standard that the relevant people can work to and are interested in is fantastically hard work. It's not just a case of writing something down and say "Agree to it!", everything has to be discussed, specified and of course there is always a slight worry in that you might be heading in the wrong direction etc.

Lack of apparent interest from vendors is also somewhat discouraging. There are quite a few specs up on freedesktop.org that are only implemented by GNOME, with KDE "pending". Then for instance Mosfet comes along and claims the thumbnail spec is stupid for reasons x, y and z and proceeds to invent his own (the so-called "professional" thumbnail spec) and ask for it to replace the existing one! Not exactly encouraging.

Interoperation adds overhead and complexity

Interoperation means passing data between two different programs over some common bridge. This means typically writing some sort of connector code. In the best case, someone is able to bundle that connector cod into a library.

Consider coding to a web service interface such as SOAP vs. just keeping your application within one programming language and using its built-in constructs for passing data. With the web service, you either have to marshal into and out of SOAP envelopes on your own or use a library. However, not all libraries themselves interoperate. Hence, you have to test for compatibility by running against a suite of other implementations, all of which are also supposedly standard. It's the browser wars all over again. If you don't bother with interconnectivity, the job is over more quickly.

To get an interoperability standard that you can just code to seems to take the developer community years of effort. Yes, there is value to interoperability, and that is why people do actually undertake things like web services projects and spend years trying to develop standards. But for a first project, or even a mature, successfully functioning product, interoperability is not likely to be a first instinct.

Worse with proprietary software

I agree that it's difficult getting open source projects to interoperate. I think the problem is that interoperation is hard, often harder than developing a program that works in isolation. Writing a simple mail server would be easy, you could build it on top of something like Distributed Ruby [rubycentral.com] and have it working in a day. Writing a mail server that interoperates properly with everything else that's out there is a totally different proposition.

Whatever the situation with open source, it's far worse with proprietary software. No open source project that I'm aware of has anything as difficult to interoperate with as .DOC.

Cooperation

> Examples of successful interop projects include
> freedesktop.org, the cooperative effort between
> GNOME and KDE."

And Xfce [xfce.org], too.

But on the aspects of cooperation between Software Libre projects, the reasons are fairly dead-on. It's not easy to cooperate with someone when you aren't forced to. Hell, I can't even cooperate with myself somedays.

Closed source software

You might as well ask why doesn't closed source software interoperate.

In a perfect world everyone would write software that works together. One shouldn't only look at getting open source software to support a common standard, we should try getting everyone to support common standards.

It's not my expirience that getting open source software is all that hard. Getting non open source stuff to work with anything is hard. Closed standards is want gets in the way of interoperability. With open source you always have the option of adding the features you need. If two open source applications can benefit from working together, it will be implemented by someone who needs it. The original authors might not always see the need for a given feature.

I don't see it as a problem in the open source community. It's fare worse is commercial closed source software.

Obligatory quote

The great thing about standards is that there are so many of them.

I think the real issue is a slightly different one

I'm not at all sure that OSS does interoperate poorly. I would rather ask:

Why, when there's a need to interoperate, does OSS invariably fall back on the 'chain of programs communicating via a pipe of characters' model from the 1970s, even though mechanisms for defining rich, concurrent interfaces have been in common use for ages everywhere else?

I know there are many good reasons why pipe'o'ASCII software projects do what they do. I also know that projects like KDE have made considerable steps in what I think of as the right direction. But the lack of componentization and well-defined interfaces in Linux-style software is one good reason why I'm glad Microsoft (and even -- yechh -- Sun) still have a strong role in keeping things moving.

(loud crashing sound as post is modded down for not being unix-centric enough)

The Other Side of OSS development

A lot of Open Source projects are done because the primary developer(s) want something that is either not readily available in existing software (the original mantra of OSS) or they want certain things "their" way. Some developers are not even aware that a standard exists for what they are trying to do and will do it the way they think is the best while other developers would care less about the standards. Its is important to create awareness about standards and their importance (believe me, lot of developers don't understand the importance of standards and think of them as unnecessary burden). When a project idea comes to the mind of a developer, a lot of times the last thing a developer will think about is existence of any standards. Like the article described, ego and NIH syndrome also is a factor. The mentality is also that "if they developed standards, let them develop the product too. I will do it MY way." Again, this boils down to understanding the importance of standards and implementing them in your projects if one exists.

- Jalil Vaidya

and I suppose...

And I suppose Microsoft "interoperates" better?

The premise is wrong. Interoperating is hard, often not worth it, and sometimes downright bad. But to the degree that anything interoperates, open source probably generally does a whole lot better than proprietary software.

Integrates fine

I find open source tools like perl, bash, grep, emacs and so on integrate well. I can process any text file I want!

After over 20 Years

I think Unix/Linux integrators keep frittering away opportunities, since they don't want to really standardize on a directory layout/interface for common utilities. I think some of this happens due to laziness (two competing versions are implemented concurrently but the authors don't unify their interfaces), but as djb says there is a short term local reward for fragmenting the interface [cr.yp.to] that the integrators choose in spite of the long term global penalty. This has held Unix/Linux back in my opinion, since code developers and administrators hate how these incompatibilities make it difficult to configure/install software properly.

What the heck?

Open Source doesn't conform to standards?

DNS?

Sendmail?

These aren't standards compliant?

And now you're going to tell me WINS and Exchange ARE?!

Perhaps the problem is not that "open source software doesn't conform to standards." Perhaps the problem is that "modern software considers itself too good for standards," which is entirely a different problem and isn't open-source specific.

-JDF

The reason is simpler than some are making it out

I've seen some posts that tap-dance around the real reason that OSS in particular doesn't interoperate well, but they all seem to miss the mark slightly. It's not really about ego, or the NIH syndome, or laziness, or poor design (if any).

The real reason, I believe, has to do with the fundemantal drive behind an open-source project -- find an itch, then scratch it. OSS projects (in general) start because someone sees a specific need or want for software that performs a specific purpose. By its very nature, that project is looking at the world in a bottom-up fashion -- and that inevitably pushes interoperability off until "later".

Integration or interoperability is typicaly an "add-on" to an already successful project -- no one really starts thinking about "well, I'd love to be able to do X from program Y" until both projects X and Y have developed strong user bases. It's sort of a natural selection in software -- the "best" projects survive and eventually breed (interoperate) with other projects to evolve higher-order software.

Of course, that's just my opinion. I've been known to be wrong... though of course, those who discover that have been known to disappear...

Are we helpless?

I thought that was the reason Perl, Python, and sockets programming are so popular.

This is a non-starter for me; I've never been hampered by interop problems with open source software.

Authors of open source start out to solve their problems, not mine. I can solve my own problems with readily available tools.

It's because...

... if I'm starting a new project, it's usually because what exisits already, whether a standard or not, just doesn't work for me. Now I may or may not incorporate standards into a project depending on what I need and whats available, but I may not.

Now I'm not working on any grand applications, and I'm not trying to solve the worlds computing problems, just mine. I suppose this could be loosly defined as an ego issue, but I assure you I don't think it's because I'm better then anyone else, it's just because I want something to work differently or work in some specific way.

I would imagine that the majority of open source projects start ou this way. That said, I think there are a number of open source projects (more and more lately) that do adheare to standards more so infact then many commercial projects, it all depends on where you look. The nice thing about the open source stuff... if you don't like it, rewrite it, or use something else, or pay someone to create what you want if you are so inclined... it's all about choice.

It's quite simple, really

The main reason standards are not or only poorly supported are:

1. the standard is not freely available (MPEG, all IEEE and ISO standards)
   
2. the standard encompasses every other standard under the sun, and nobody can possibly support it all without licensing all the crap from everyone (this is actually quite typical these days. 100 companies come together and define all their combined "intellectual property" as an "industry standard" and thus force everyone else to license their crap from them. This is what happened to MPEG-4, and to an even greater extent, MHP (set top box standard, includes DVD, MPEG-2, Java, and much more))
   
3. the standard sucks because it leaves ambiguities or simply assumes a lot of stuff that is not stated clearly. This is most often a problem of incompetence, because almost nobody is good in writing standards. You have either technical people who know what they are talking about but assume too much to write a good standard, or you have the technical writers who don't assume anything but don't understand what they are writing and thus produce absolutely unusable documents. This is an increasingly bad trend in RFCs, unfortunately.
   
4. The standard is bad because it was written by an utter moron. This happens if marketing companies are trying to participate in a standards process because they think it will give them more credibility. The results are desastrous.
   

Almost nobody is trying to implement standards badly. But some standards are so bad that you can't implement them without getting a brain tumor. Just have a look at tmpfile() in POSIX, for example, whose semantics make it impossible to use it without creating a race condition. Or look at DVD, which references a hidden trade secret called CSS, which is not part of the standard, but you can't be standard compliant without CSS, which forces you to sign sinister contracts with the content industry cabal.

Who does all of the sh*t work?

I have never participated in an open source project so I can't say but I have worked on projects for pay.

Supporting standards is a lot of work and most of it is not particularly fun or intellectually rewarding. It is just a pain in the butt. Who want's to do that for free?

Once the code is "complete" (code is never complete) it must be tested to insure that your implementation of the standard will work with other implementations of the standard. This testing is tedious, time consuming, and diverts resources from other parts of the project. You usually also have to have programmers who worked on each implementation available so that they can work out any inconsistencies. I've done this before for pay and I still didn't want to do it.

Another thing that always pops up is whose implementation is more standard? When an inconsistency arises one side has to make a change. I've gotten in the middle of pissing contests with programmers who each insist that the other is wrong and they aren't going to change their code to work around their bug. What fun!

I know that there are a lot of OSS developers out there who take their work very seriously and put out the highest quality product but they also have day jobs, lives (I hope), etc. that compete for their time. If I was doing the work on my time, I would probably tend to do the stuff that I found more rewarding. Things like rigorously supporting standards and all of the sh*t work that goes along with that would probably take a back seat to working on a new feature or something else that I considered challenging and exciting.

Unix Pipelines

Almost all of my open source software interoperates. It's the interoperation we call "unix", and which is so graceful and transparent we don't even realize they're different programs using the same rules to pass information around.

Shareware on a Mac

For the sake of discussion, here's what I think.

The reason I'm seriously contemplating switching to OS X is because they have shareware that is worth paying for. I don't mind spending $10 on a nice editor or a utility that saves me some time.

On linux, there are a lot of diamonds in the rough in terms of software, but majority is... well, incomplete. I can't complain, because it's free, and because who am I to say how people are to spend their time?

What is true, however, is that there is no motivation for a hobbyist to follow standards if he doesn't want to. If however, there is at least a $5-$20 contribution from the users, then there is at least some weight to the user's request. An honor code if you will.

Simply put, I'm starting to see where shareware might fit in this world, and it's basically a way to transition a hobbyist project to something more mature and useable.

-Nikita.

Think of it as evolution in action

When standards exist, Open Source projects are frequently better about following them then closed source projects. But not necessarily. If the developers consider that the standard is really broken, then they'll ignore either it or parts of it. The only difference from closed source projects is that they won't break the standard in order to keep you from working with something else.

When standards don't exist, nobody complies with them. If something is patented, then it obviously can't be standard in any meaningful sense, unless the patent is freely useable for all standards conforming uses. Some "standards" bodies don't seem to understand this.

And when personalities come into conflict, all conformance can go by the wayside.

If I look over this list, the Open Source software generally comes out at least as good at standard conformance as the proprietary software. If for not other reason, because if it's easy to put standard conformance into a project, it's reasonably easy to retrofit it in. And features tend to not get removed (though they can be turned into compile-time optional features).

Over cycles of iteration, Open Source software either becomes standard conforming, establishes a standard, or becomes irrelevant. (I.e., evolution in action.)

E.g., would you rather try to read MSWord documents with OpenOffice, or to read OpenOffice documents with MSWord? MSWord has a positive benefit to MS with breaking adherence to the file format used by the previous version. It causes people to buy the new one. OpenOffice has no such benefit to the producer, and this is a benefit to the user.

Someone explain bash

I've read a lot of comments here, with some interesting points, excuses, or disagreements on the premise that OSS doesn't support standards.

Bash supposedly conforms to Posix standards if you invoke it with the --posix flag. (Why it shouldn't default to posix-compliant I don't know) However, bash is not compatible with /bin/sh. WHY???!!!!

Will someone tell me why bash is the ONLY /bin/sh-like shell that I can't `echo "hi!"` in????

In case anyone things I'm just ranting (I probably am ranting, but that's not all there is here :-), consider my question closely. sh, ksh, zsh, ash, and every other shell that uses sh syntax (i.e. not the csh variants) deals with the above statement in the same way. Bash doesn't.

Why would OSS deliberately develop a shell (the default universal Linux shell no less) that breaks such a fundamental and long-standing de facto standard?

some kind of joke?

"cooperative effort between GNOME and KDE"...
one worrd for you RMS

Successful example?

Since when is freedesktop.org a successful example? What did they achieve yet besides specifications?

Development standards base.

We have Linux Standards Base -project, maybe we should also have Linux Developers Standards Base -project which defines how you should use/develop libraries and plugins in your projects so you can interoperate with multiple standards.

So if you develop a Mail User Agent it could read all of the mailbox-standards by using the default libraries for these. (technically possible? definitely. hard? harder..)

Ofcourse it should define other standards too: configuration files, commandline parameter styles.. everything that is common with software. But there should be some freedoms due to the nature of OSS-development like freedom to use any GUI-library etc.

There would be a lot of fights over the standards, yes. But it might be worth it in some cases. I can see a few problem cases with this, but it could be useful in many common situations.

The twisted part:

OSS-developers usually wants to code everything by themselves, even if there is a library that does something for you.. they still tend to code one by themselves because the library misses one feature or something. Just add the feature to the library, don't create your own! (From this we will come by the problem of library developer not accepting your addition to the library... try to discuss about it.. why it is useful addition etc.)

OSS-users usually wants to use software that doesn't have a lot of libraries. Have you ever begun to install some software and noticed that it is going to install 20 libraries in addition? Scary huh? We should change our mindsets, repeat with me: "Libraries are good, libraries will liberate us."

Answer by Fortune

Remarkably, the /. fortune at the bottom of the page had the most succinct description of the problem:

I must Create a System, or be enslav'd by another Man's; I will not Reason and Compare; my business is to Create. -- William Blake, "Jerusalem"

But it does interoperate.

I don't know what barbarian tools these guys use, but the software I use follows standards well enough.

Mozilla
Sendmail
Gaim
Gimp
Gcc
LibXML
Apac he

I'm sure the list could grow and grow.

OSS Interop Standards Needed

Disclaimer/Invitation to Flame: I am not a coder by trade. Rather, I am a writer/UI planner/designer. I can hack PHP, PERL and SQL/MySQL when called on, but it is not my natural skillset.

OK, with that out of the way, here are a few thoughts.

1. Open Source Software (OSS) is growing and gaining market share everyday. So, the objective of this discussion should not be is OSS good or bad, but how do we remove roadblocks to it's inevitable success while enabling long-term sustainability and avoiding the bloat that so cripples many commercial platforms (ex. Windows being the obvious target).

2. To paraphrase the Bard ... Discipline must be used. (Capt. Flewellyn, Henry V).

As several folks have noted, standards need to be agressively developed and enforced. I do not hold the opinion that OSS dev is complete anarchy. Far from it. Some of the most elegant software I use springs from the OSS Community. And, not coincidentally, most of these bits of software use standards (XHTML 1.0, CSS 2.0, Blogger API, etc.) in addition to those already mentioned (Sendmail, etc.).

At the same time, I would have to agree with the position taken by Gregor and Paul that perhaps, some of the tribal culture of OSS dev teams creates unneccesary isolation and, as a result, creates many missed opportunities for a bit of the "blue sky" thinking that enabled disparate groups of people to build on one another's work to create the "next big thing". For a great example of what I am talking about, read 20 Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable [oreillynet.com] by Marshall Kirk McKusick. The link provides an excerpt anf the entire essay can be found here ... Open Sources: Voices from the Open Source Revolution [amazon.com].

But, I digress.

Because we have so many bright dedicated folks working in relative isolation, we spend lots of cycles on redundant work. So, I think we need some of those bright minds working on, for lack of a better term ... "OSS Interop Standards". Said standards could and likely should be subdivided by use case (web/desktop/server) and perhaps technology/language. Though, the risk there is that too much granularization will kill interop, which is the point in the first place.

Certainly, the big boogeymen of developing such standards are stagnation and the death of creativity. And, those are certainly legitimate concerns. However, they are only concerns if the standards are not viewed as a malleable, living document. The idea is to steer development towards greater, platform/language-independent interoperability.

The blogging world already sees the light (Blogger API and the emerging MetaBlog API). By enabling these standards, Moveable Type can talk to Blogger and, in the near future Xaraya. Some will argue that programming this way makes it too simple for the end-user to jumpship from one piece of software to a competitor. (ex. Moveable Type from Grey Matter). While that may be true, this completely misses the point and the opportunity. Conversion scripts/apps will exist no matter what. That is a fruitless fight.

By using Interop Standards to enabling seamless data exchange and even easy conversion from one OSS tool to another, developers ensure growth and distribution of their products, particularly with in the less technically savvy design and end-user markets of the web. If trained as a coder, it may be relatively simple to jump from PERL to PHP to .NET.

But the folks that are ultimately going to use free products (Moveable Type) or resell pro products (Moveable Type Pro), the jump is not so simple. They are far more likely to latch on to one language (in my case PHP and a shade of Perl).

Selection/Drag-n-drop

What I think would help a lot is if there were more standard conventions for selections. This would make it possible to enable more drag-and-drop features across programs. And using drag-and-drop in, say, GTK, is really quite easy (easier, I believe, than in Windows).

For example, one piece of code I'm working on uses images and palettes; and it would be really nice if I could drag one of my palettes into the Gimp, or one of the Gimp's onto the palette window in my app. Same thing for individual colors and images.

Is there a group or project trying to set standards for selection at the application level? Perhaps this should be an extension to the X drag-n-drop spec.

No evidence

This article doesn't prove its own premise. Where is the evidence that open software is less likely to implement a standard than closed software? I can think of so many counterexamples, because OSS tends to actually define the standard. See: every major internet protocol.

This does not have to be the case...

I wrote a ftp API an a network client located on http://j-ftp.sourceforge.net

It is GPL, but it is stable and even javaworld has an interesting html table comparasion (link on the homepage) which shows that it is superiour to most apis out there in many cases, because no api implements *all* standards.

It is that good because it was designed completyle following the rfc and is imho fully rfc 959 compatible. That took its time, almost 3 years now, but i don't think the api could not be used in a productive environment.

just my 0,019725 euro ;)

Surprising? I think not.

"There is an interesting article on Advogato on why it is so difficult for Open Source projects to interoperate or support common standards. ...

This is something of an extrapolation, IMHO. The referenced article described a "sprint" - what we old-timers remember being called a "workshop" ;) - that brought together people from various OSS projects working in the same general problem field to find out how well they could work together and benefit from each others' knowledge.

Put like that, it's perhaps not too surprising that the workshop was useful. I'm not being sarcastic here: I think the organisers deserve kudos for recognising an area where cooperation was feasible and taking on the responsibility (and associated hassles and risks) of organising the event with fellow interested parties. Acks also to the sponsors for making it possible, and the more such events the better.

Now, if someone could come up with ways for people to communicate productively without meeting personally and without being predisposed to cooperate in the first place... well, I'd suggest you keep quiet unless you want to be nailed to a tree to die of thirst.

Open Source...

I had been doing an open source project recently for the past few months. I never really attracted any users. I only know of one person who got it to compile and work.

What really perplexes me are the projects which have managed to get people interview them and discuss the project, and yet after 3 years they still have not released anything.

Or those projects that existed for about 10 blips of a microsecond, got lots of hype and coverage, but you can not get them anymore. They are gone.

I at least did what I said I was going to do and it works. Maybe it doesn't look pretty (there is no such thing as an Open Source Artist people). Maybe it is hard to use (WTF do you expect for free? Blood?). Maybe it's a total hack job. But I did it and at least I can walk away from you all feeling doped out buku-good.

The problem is you people expect this stuff to rival professional software on Windows, and yet maybe 1% of you actually understand it is hard work and takes alot of time. Most of us are not getting paid to do this. And when you get cold shouldered, ignored, and flamed by morons about how the program sucks, it doesn't help very much.

I will continue to work on Open Source but I am not trying to impress people nor do I care whether they use it or not. It is open because I know that someone out there might be able to benefit from it who can't afford the pro stuff, and appreicate it, and for that one person, that is who I am working for.

Whatever projects I find useful I always write to the developer and say thanks. Why? Because I've been in there shoes. 99% of them are doing it for free. They do it because they want to do it. They do it because they love it. That is why I do it. That is why I will keep helping Open Source projects, and making more code.

Until you actually have to work hard for what you have you will never appreciate what you have.

The reason Open Source is so diverse and uninteroperable is because it encapsulates the spirit of the whole movement to begin with. It began as the pet project of a bunch of people who wanted to hack on code. They do it because they like doing it. Now it's a clique thing where it's cool to be a part of it. When it first started (and I've been in and out of this scene for 8 years now) nobody knew what open source was about. Linux? What the fuck is a linux?

The system was funky ass and I remember blowing up my monitor trying to get X-windows to work. Yeah it was not anything like what distributions are like now.

The reason the state of Open source is not rivaling the stuff on Windows is because 99% of you expect it but don't want to put the hard work into making it happen.

The Spirit of Open Source is the lone hacker against the system. Hacker's hate standards. They hate coding conventions. They hate shit that is closed and they can't tweak the code. They hate whiney users who are to lazy to figure out how to work shit and won't type man. The whole spirit of it is to make people get off their lazy ass and dig into things and find out how they work. It's not and never will be a plug-n-play system where everyone gets along and everyone speaks the same language and looks the same. It isn't going to happen. It goes against the whole philosophy of what the system is about: being free, rebelling against the corporate bullshit that chains you down and makes everyone write code the same way.

If Linux becomes some standardized, plug-n-play, corporate system, I will find some other operating system to use and be a part of. I come to Linux specifically to get away from that very shit.
I like Linux not because I think it's user friendly but because the people making things are Linux are truly innovative, and yes, it is much harder to get working than Windows, but it's worth it, because it is far more flexible and configurable. But it takes work to figure the stuff out.

I'm not against people taking the system and using it to make money with, or making distributions that do all this stuff for

Teamwork Requires Standards

Bringing this conversation full circle ... One important thing I think some folks are missing about the standards argument is that no one is suggesting the OSS be legislated to use any given standard. As folks have mentioned, that flys in the face of the OSS ethos. If lone devs want to work in isolation, then have at it. Some of the most brilliant men in history have worked as hermits. This discussion is not about making the lone wolf feel like he should "get with the program". Rather, the discussion focuses on making things simpler for teams of OS devs. THe Team for Product X requires standards against which they should develop, else none of the code will fit together. We all naturally do this when working in a team. All that is being suggested is that Standards are required for Team X and Team Y and Team Z to connect their individual products in meaningful ways while not reniventing the wheel. If your product can or should operate in complete isolation, don't adopt the standards. No harm, no foul.

Re:Open Source

Actually ximian has done some really nice things with openoffice to improve interoperability....
Although they mostly focus on gnome it should also interoperate better with kde since with the freedesktop.org stuff things like cut and paste are becoming less of a problem...

XFree86 is fine for desktops, just don't expect it to be a top gaming environment yet.

Jeroen

Re:Open Source

Quote:OpenOffice + KDE + XFree86** != Productive Desktop Environemnt (for example).
Really? I seem to do just fine with this arrangement. In fact, I would say that I am more productive with open source programs than I ever was with proprietary software.

Re:Open Source

Like Microsoft and Windows XP

This then is the meaning of the /. Bill Gates==Borg icon? The company and its product, all wired for sound?
Look at things like Perl and databases and tell me again that things aren't interoperable.

Re:BFD

"I can display an X11-compliant program running anywhere on any X11-compliant display running anywhere."

Not necessarily. Some X servers don't support certain X extensions and/or don't come standard with certain fonts the program is expecting. That'll cause behavior/display to be less than certain.

And you ARE aware that Windows XP (possibly 2K, I think) comes standard with a remote desktop viewer program, right?

Re:Standards

As much as you guys may hate Windows, the beauty of its "standards" is that I can take just about any program from the Windows 95 era and run it "out of the box" on a modern Windows XP system.

No, you can't.

Remember all that Windows software that worked on '95 but failed on NT? Good luck getting it to work on XP now.

Eventually, Linux and other Open Source ventures are going to have to "bite the bullet" and learn to enforce standards.

We're already there.

GNOME developers, for instance, *do* have a registry they can manipulate in a standard way (GConf!), a "start menu" they can manipulate via standard means, and a web browser which will render things in a standard way (gtkhtml). We have a directory layout standard and have for quite some time (yes, it's a flexible standard -- folks can, at their discretion, install apps under /opt/ or use package management systems like STOW, but that's fine; it hurts no-one, as long as the app has the minimal intelligence needed to deal with being installed under an arbitrary root and a system for locating libraries such as pkg-config).

Sure, there are Linux installs that aren't also GNOME installs -- but what does that matter from the position of evaluating GNOME (not Linux, but GNOME) as a desktop environment and development platform?

Re:Open Source

The only times things do work is when they are centrally controlled, but that defeats the ideals of Open Source (Like Microsoft and Windows XP, well integrated).

I completely agree, this is the only way things can go right, a (one) company that creates its own standards will always be compatible with itself.

Obviously the different prints and formatting you have while opening files on different machines are features you do not have when all the machines run the latest patch version of the latest full install (except sub-option X in window Y, otherwise you're in trouble).

Yes, Microsoft _is_ the standard, so they must always be right.

/me gets struck by a bolt of electricity from my power supply

Re:Standards

Gotta love Slashbots -- "oh shit, he's right about Linux sucking in some regard! Better mod him down."

Re:the old saying..

I thought it was:

The nice thing about standards is that everyone can have their own.