Forgot your password?
typodupeerror
Caldera Programming IT Technology

ESR to Shred SCO Claims? 554

Posted by michael
from the woodchipper dept.
webmaven writes "According to this article in eWEEK, ESR has released a utility called comparator for analyzing the similarity of source code trees. The technical details are interesting, in that ESR says he is using an implementation of a refined version of the 'shred' algorithm, with higher performance (on machines with enough RAM) than other versions. ESR won't say whether he intends the comparator to be used to compare older Unix code to Linux so as to be able to refute SCO's claims, but it's obviously well suited for such a purpose. Interestingly, as the shred algorithm can run reports on source trees using only the MD5 signature shreds (once generated), it is possible to use it to compare trees without direct access to the source code itself, leading to a possible use in comparing various proprietary source trees with each other and with Freely available code bases such as Linux and *BSD without requiring actual disclosure of the proprietary source code (a neutral third party could generate the shreds on a company's premises, and leave without taking a copy of the source with them). I'll be interested to see if (or which of) the proprietary vendors allow their source trees to be 'shredded' for such comparisons, and whether this becomes a standard forensic technique in source-code copyright and trade-secret disputes."
This discussion has been archived. No new comments can be posted.

ESR to Shred SCO Claims?

Comments Filter:
  • maybe... (Score:5, Funny)

    by b17bmbr (608864) on Tuesday September 09, 2003 @04:54PM (#6915153)
    microsoft can just shred their source tree and start anew. maybe...
    • Re:maybe... (Score:5, Interesting)

      by jmv (93421) on Tuesday September 09, 2003 @04:58PM (#6915194) Homepage
      Actually, combine this with the "shared source" program from MS and it would be easy to see if MS did (or did not) copy GPL code into Windows as some suggest.
      • Re:maybe... (Score:5, Insightful)

        by fireboy1919 (257783) <rustyp@freeshell.COBOLorg minus language> on Tuesday September 09, 2003 @05:48PM (#6915715) Homepage Journal
        Right. Because as we all know, people who pay Microsoft the huge bag 'o money that it costs to see their source are primarily interested in the pursuits of OSS to see if Microsoft has copied anything it shouldn't have. And Microsoft's NDA surely gives them the right to do this.

        If anyone is able to prove Microsoft is doing something illegal via the shared source initiative, they'll probably have to do it illegally.
        • Re:maybe... (Score:4, Insightful)

          by toast0 (63707) <slashdotinducedspam@enslaves.us> on Tuesday September 09, 2003 @05:56PM (#6915768) Homepage
          If you've licensed code from microsft, and it turns out to be GPL, the license under which you got the code is invalid, so it wasn't illegal to determine if they improperly took code.

          On the other hand, if all their code checks out, testing for that may violate their NDA, but it'd be difficult for them to show you checked their code if you don't mention it.
        • Re:maybe... (Score:5, Insightful)

          by Courageous (228506) on Tuesday September 09, 2003 @08:31PM (#6916812)
          And Microsoft's NDA surely gives them the right to do this.

          A term in any contract, including any NDA, as stipulated by any party, which would obligate the other party to not report a violation of law, either statute or criminial, is PER SE unlawful and cannot be enforced within any jurisdiction of of most first world countries. Any contract bearing such a stipulation would in fact be at significant risk of invalidating the ENTIRE contract, not just the unlawful provisions therein.

          C//
      • by poptones (653660) on Tuesday September 09, 2003 @07:03PM (#6916256) Journal
        This entire argument is happening for ONE reason: various governments of the world )specifically, in this case, the US) has afforded COPYRIGHT protection to works that contribute nothing to "furtherance of the state of the art" and nothing to "the progress of science." If I build a power saw, I can patent unique aspects of its design but have to reveal those aspects.

        Copyright is misapplied to source code. Either REVEAL THE SOURCE or you only get protection on that which you "publish" - namely, the binary.

        Put up or shut up; no source, no copyright on the source. You won't share it, you don't need it protected.

      • Actually, combine this with the "shared source" program from MS and it would be easy to see if MS did (or did not) copy GPL code into Windows as some suggest.

        More importantly, get something like this accepted in a court of law as a legitimate way to do an initial assessment of code yet still preserve a litigants right to code privacy, and you are going to have not just MS but a number of big companies shaking in their boots. Not necessarily because they did steal anything but because they have to realiz
    • Re:maybe... (Score:3, Informative)

      by Anonymous Coward
      look how Microsoft is directly trying to bias the case more with onesided biased news: check out [from today]: [msn.com article from supposed tech analyst Jonathan Cohen [msn.com]

      then read:
      More on Jonathan Cohen [threenorth.com]

      Microsoft MSN a biased propaganda machine. Only shows one side of the facts (the lies).
  • SCO! (Score:4, Funny)

    by scovetta (632629) on Tuesday September 09, 2003 @04:55PM (#6915156) Homepage
    Of course, we can just trust SCO to show the right hashes. Why would they lie?
    • Re:SCO! (Score:3, Interesting)

      by jmv (93421)
      Ths think is that the hashes could be generated my any organisation that has access to the SysV source code. There are many of them (IBM being one).
    • Re:SCO! (Score:5, Insightful)

      by mik (10986) * on Tuesday September 09, 2003 @05:05PM (#6915300)
      The point is that we don't need SCO to do anything. Presumably any of the many people with legal rights to SCO source code can publish the hash list without divulging any of SCO's (ahem) "IP". Even more interesting is the theoretical possibility of comparing historical releases of SCO trees against GPL-licensed code, thus (perhaps) demonstrating that SCO has illegally violated the IP of OSS developers. Of course, hash comparisons alone would be unlikely to convince a judge/jury of anything. They ought to be sufficient grounds for some embarrasing subpoenas, and maybe some really neat cease-and-desist orders, though.
    • What if...? (Score:5, Insightful)

      by bladernr (683269) on Tuesday September 09, 2003 @05:34PM (#6915604)

      What if this ESR tool runs and finds commonality, and the research shows that, in fact, SCO's rights were breached. Remember, this type of analysis is a two-edged sword. The purpose of this ESR is to remove doubt... but remember doubt could be removed either direction.

      So, given that hypothetical, what would people here think? Would you forive SCO? Would you concede SCO's point, but think that SCO defended their rights in a very poor manner? (this, btw, is what I would probably do). Would you stick your fingers in your ears and refuse to accept the outcome, and believe in some vast -wing conspiracy?

      Obviously, the Linx movement would carry on. I don't think the death of Linux is even worth discussion. Some recourse would happen, probably monetary damages, and the offending code would be removed.

      My real curiosity is how people's attitudes or feelings would change (or not change) if it turns out SCO is right (however unlikely that is).

      • Re:What if...? (Score:3, Insightful)

        by good-n-nappy (412814)
        It still would not prove SCO's point. They have to answer the question of why they distributed the Linux source even after finding out it contained their valuable "IP."

        Most of us are relying on common sense and don't really care whether a few lines of archaic code were copied. Given SCO's

        1) previous sales of Linux
        2) misinformation about owning Unix
        3) waffling on what IP is violated
        4) refusal to show copied code
        5) frequent, inconsistent press releases
        6) heavy insider trading
        7) ridiculous licensing terms
        8
      • Re:What if...? (Score:3, Insightful)

        by The Cydonian (603441)
        A two-cent observation regarding most media "debates":- folks usually attack other people, not positions. It's usually very rare to find anyone who's changed his position based on new evidence or inference. Not impossible, just rare. (Which, in a way, is why not getting emotional about anything is always a good idea; that way, you can base your support on rational thought, not objects)
  • by More Karma Than God (643953) on Tuesday September 09, 2003 @04:56PM (#6915173)
    If there is, why couldn't MD5 shreds be used as a lossy compression scheme for code?
  • by Anonymous Coward on Tuesday September 09, 2003 @04:56PM (#6915175)
    This will only serve as another black eye on the Open Source community. ESR should know better that to shred SCO material prior to a trial.
  • by BlackBolt (595616) on Tuesday September 09, 2003 @04:56PM (#6915178) Homepage Journal
    Did he write it in Python? And did he complete it in under 6 hours?
  • Doubt it will help (Score:5, Insightful)

    by Brahmastra (685988) on Tuesday September 09, 2003 @04:57PM (#6915190)
    I think the question here is not about whether there is common code between SCO and Linux. There is no doubt that there will be common code because of the common origins. The issue here is that SCO does not own that code.
    • by djh101010 (656795) on Tuesday September 09, 2003 @05:00PM (#6915234) Homepage Journal
      If there's going to be a line-by-line comparison, this is the tool to do it. Once those lines are identified, *then* it's simply a matter of finding out the origins of them; that's where we can roll it back to a textbook published in 1973 or whatever.

      Until the lines that are common are identified, it's impossible to defend against the accusations. Because of that, I bet Darling Darl won't allow it to be used. The question is, how to turn the inevitable refusal into something that shuts him (up|down).
    • by Azog (20907) on Tuesday September 09, 2003 @05:37PM (#6915635) Homepage
      Well, this would still help determine what the common code is.

      If ESR is given the big list of MD5 sums of SCO's kernel by someone who has legitimate access to it, and he runs his shred tool to compare it to the Linux kernel, and a bunch of stuff turns up matching (as expected) he can still see WHAT was matching because he has the Linux sources.

      So then he can look at that and say, "hmmm, it looks like part of this ethernet driver is the same, and this NAT implementation, and bits and pieces of the VFAT filesystem code..." and then, find out how those got to be the way they are in Linux.

      If it can be proved that the matching code is totally legit in Linux, (which is what I would expect) then it follows that either (a) SCO actually stole stuff out of Linux, rather than the reverse, or (b) Linux and SCO both took the code from a third source, like BSD.

      Otherwise, option (c) is that Linux actually contains code from SCO which it should not. But this is still an improvement on the current situation, because it would allow the Linux development team to FIX THE PROBLEM.

      Either way, (sooner or later, depending on if Linux fixes are required) it will shoot SCO's claims so full of holes that any reputable journalist reporting on SCO's latest insane claims will have to mention that "... but the source code has been analyzed and all code in Linux similar to SCO's software has been shown to be completely legitimate...", or "... but all code in Linux which SCO might have had a valid issue about has been removed..."

      SCO's big stick right now is FUD. Fear, Uncertainity, and Doubt. The shred tool can remove the uncertainty and doubt. Only SCO will still have the Fear. :-)

  • Nah... (Score:4, Insightful)

    by SargeZT (609463) <pshanahan@mn.rr.com> on Tuesday September 09, 2003 @04:58PM (#6915195) Homepage
    This shouldn't be relied upon in the court of law. Although I acknowledge that SCO likely has no IP claim over Linux, it should have a fair case. A program that would rule out code similarities does not rule out code that is based on the SCO code. There are hundreds of ways to do a single thing, and if the GNU/Linux took ideas from the SCO kernel, SCO may be as eligible for compensation as if it were directly copied from SCO.
  • by Teahouse (267087) on Tuesday September 09, 2003 @04:58PM (#6915196)
    The truth is out there, we will finally get to it without signing a SCO NDA. This should end the case before it begins. SHRED ON!

  • by TexVex (669445) on Tuesday September 09, 2003 @04:59PM (#6915216)
    This just in. SCO to sue ESR for patent infringement over "comparator", a software package that performs comparison between different sets of source code to determine if any code is copied between them.
  • Other uses? (Score:4, Interesting)

    by Not_Wiggins (686627) on Tuesday September 09, 2003 @05:00PM (#6915235) Journal
    It might be interesting to see how different families of Linux/Unix compare... maybe generate a veritable "family tree" of relationships.

    Of course, that also depends more on how differences are actually calculated. Still, could make an interesting project to relate OSes based on how much shared code they still retain and show it in a graphical tree format, ala "family tree." 8)
  • Who cares? (Score:3, Insightful)

    by Otter (3800) on Tuesday September 09, 2003 @05:02PM (#6915262) Journal
    OK, I admit that a) the guy annoys the hell out of me, b) his yapping about "one of us" DOS'ing SCO is yet another case of him embarassing Linux while aggrandizing himself and c) just the quotes in this article alone make me want to slap him. So if someone else had been involved with this, I probably wouldn't bother to care.

    Anyway -- who cares? There's no question there are plenty of common chunks between Linux and SCO-owned source. And that there are ways to find them. The question is what they are (which SCO isn't saying) and what their common origin is and where that origin falls in the murky history of the Unix codebase. It's not as if anyone has been saying, "We're helpless in the face of this computational problem. If only there were a way to compare large bodies of text for common elements!"

    Never mind that there are probably people who can compare both codebases in their heads.

    Maybe he's made some major algorithmic breakthrough. (I doubt it but, but I'll leave that to the experts.) But this story is just him yapping again.

    • Re:Who cares? (Score:3, Informative)

      by jmv (93421)
      I think the difference is that a 3rd party that has access to the SysV source can compute the hashes and make them public without violating copyright. That way anyone can look for common lines with Linux and see where they came from (legal or not).
  • by Malfourmed (633699) on Tuesday September 09, 2003 @05:02PM (#6915267) Homepage
    The Sydney Morning Herald continues its mainstream coverage of the SCO vs IBM roadshow by posting an article where Dr Warren Toomey, a Unix historian, says that SCO may not know the origin of their own code [smh.com.au].

    Article text follows:

    SCO may not know origin of code, says Australian UNIX historian

    By Sam Varghese
    September 9, 2003

    More doubts have been cast on the heritage of System V Unix code, which the SCO Group claims as its own, by an Australian who runs the Unix Heritage Society. [tuhs.org]

    Dr Warren Toomey, now a computer science lecturer at Bond University, said today: "I'd like to point out that SCO (the present SCO Group) probably doesn't have an idea where they got much of their code. The fact that I had to send SCO (the Santa Cruz Organisation or the old SCO) everything up to and including Sys III says an awful lot."

    He said that even though SCO owned the copyright on Sys III, a few years ago it did not have a copy of the source code. "I was dealing with one of their people at the time, trying to get some code released under a reasonable licence. I sent them the code as a gesture because I knew they did not have a copy," he said with a chuckle.

    Dr Toomey's statements come a few days after Greg Rose, an Australian Unix hacker from the 1970s, raised the possibility that there may be code contributed by people, including himself, which has made its way into System V Unix and is thus being used by companies like the SCO Group.

    Dr Toomey said this was one reason why the code samples which the SCO Group had shown at its annual forum had turned out to be widely published code.

    SCO was unaware of the origins of much of the code and this "explains how they could wheel out the old malloc() code and the BPF (Berkeley Packet Filter) code, not realising that both were now under BSD licences - and in fact they hadn't even written the BPF code," Dr Toomey said.

    He said that there was lots of code which had been developed at the University of New South Wales in the 70s which went to AT&T and was incorporated into UNIX without any copyright notices.

    "At that time the development that was going on was similar to open source - the only difference was that the developers all had to have copies of the code licensed from AT&T," he said.

    Dr Toomey, who served 12 years with the Australian Defence Force Academy, an offshoot of the University of New South Wales, before joining Bond University, said he had source code for Unices from the 3rd version of UNIX which came out in 1974 to the present day. "I don't have Sys V code but there are people with licences for that code who are members of the Unix Heritage Society. We can compare code samples any time," he said.

    He agreed that the codebase of Sys V was a terribly tangled mess. "It is very difficult to trace origins now. There is an awful lot of non-AT&T and non-SCO code in Sys V. There is a lot of BSD code there," he said.

    In March, the SCO Group filed a billion-dollar lawsuit against IBM, for "misappropriation of trade secrets, tortious interference, unfair competition and breach of contract."

    SCO also claimed that Linux was an unauthorised derivative of Unix and warned commercial Linux users that they could be legally liable for violation of intellectual copyright. SCO later expanded its claims against IBM to US$3 billion in June when it said it was withdrawing IBM's licence for its own Unix, AIX.

    IBM has counter-sued SCO while Red Hat Linux has sued SCO to stop it from making "unsubstantiated and untrue public statements attacking Red Hat Linux and the integrity of the Open Source software development process."

    -----

    Wordforge writing contest now open: deadline 2003-03-28

    • In all fairness.. (Score:4, Insightful)

      by k98sven (324383) on Tuesday September 09, 2003 @05:18PM (#6915462) Journal
      SCO was unaware of the origins of much of the code and this "explains how they could wheel out the old malloc() code and the BPF (Berkeley Packet Filter) code, not realising that both were now under BSD licences - and in fact they hadn't even written the BPF code," Dr Toomey said.

      The SCO Group (not old SCO) hasn't written any code in SysV UNIX.

      Anyway.. One could hope that when this is all over, the UNIX sources will be bought up from the carcass of SCOX and open-sourced, finally putting it out of its misery..

      That is, as long as SysV UNIX doesn't have more stolen code in addition to the BSD code we all know about..

      The sooner the zombie of UNIX is put to rest, the better for all the live Unices.
      • by Dr_Marvin_Monroe (550052) on Tuesday September 09, 2003 @06:38PM (#6916074)
        In all fairness, SCO's value is not in being purchased so that the source code can be freed...

        SCO's value is in acting as a totem against future companies who would try this same stunt....Their value is in their smoking carcass with Daryl's chared head mounted promanently on a high pike...

        At this point, there can be no comprimise with people who commit fraud to inflate their stock price and to promote FUD.... I believe that Daryl KNOWS that his claims are false...he deserves to fry....

        I say, "smoking head on stake" for all the SCO/Canopy group members.... leave all the execs at SCO without a job and discredited like the MCI/ENRON execs....Leave all the investors holding worthless stock certs....Somebody needs to be an example, and SCO volunteered by inflating/changing/hyping/FUDing their claims.

        I could have had a little sympathy for them if they had just filed their suit and shut-up until the trial....but at $17/share now, we need to destroy some wallets to remind everyone that it's not over till the gavel falls......
  • Be careful... (Score:5, Interesting)

    by nolife (233813) on Tuesday September 09, 2003 @05:04PM (#6915295) Homepage Journal
    The more points you discover and disprove now with SCO's claims.. the higher quality, more refined, and detailed SCO's evidence will be when this setup finally gets to a court in front of a judge. If they went to court two months ago or even today, they would have been sent home quickly with bascially easy to disprove evidence. With the help of the open source community, they are slowly changing their weapon of choice from a shotgun to a rifle.
    • by JoeBuck (7947) on Tuesday September 09, 2003 @05:33PM (#6915598) Homepage
      If we can show that SCO's violating the BSD license, maybe we can convince some BSD copyright holder to sue them first, and demand as part of discovery the MD5 checksums from "shred", showing duplicated BSD code but no duplicated BSD copyright.
    • Re:Be careful... (Score:4, Insightful)

      by Daniel Phillips (238627) on Tuesday September 09, 2003 @07:49PM (#6916516)
      The more points you discover and disprove now with SCO's claims.. the higher quality, more refined, and detailed SCO's evidence will be when this setup finally gets to a court in front of a judge.

      Having many thousands of bright minds working on our side much more balances the advantage SCO can get by snooping on our discourse, if they can even come close to following it all, that is. We outnumber them, it's stupid not to capitalize on that.

      Just think, if the word doesn't go out, there are many people who might not have come out of the woodwork to contribute their valuable input, historical recollection, interesting files, legal insight, whatever. We work in the open, we share information, we cooperate, we are many in number. They work in the dark, they trust nobody, they're afraid to ask for help, they are few. It's open source versus closed source all over again.

      Also, we each do our own thinking, we try to come up with the part we can contribute, then we go looking for the best place to contribute it. Multiply by 10's of thousands. Compare to a few fevered minds going over and over the same rotten thoughts then sending out marching orders. Seen two systems like that before? Right, it's a free market economy versus Soviet-style central planning. In the end, the free market won because it is more efficient.

      With the help of the open source community, they are slowly changing their weapon of choice from a shotgun to a rifle.

      A rifle will not help you much against a herd of 50,000 enraged penguins stampeding towards you at an average speed in excess of 100 miles per hour.
  • by Eberlin (570874) on Tuesday September 09, 2003 @05:07PM (#6915332) Homepage
    KDE GUI version should be called Krang since Shredder would obviously be used from the command line (shell). Maybe it should have helper apps called Bebop and Rocksteady. And if the need should arise, the project shouldn't fork...it should splinter.
  • by RocketRick (648281) on Tuesday September 09, 2003 @05:07PM (#6915335)
    By computing MD5 hashes of consecutive (overlapping) line triplets, the shred algorithm makes it easy to identify copied code, without ever seeing the actual code. This might be a perfect way for companies to allow a third party to compare code, without giving away any trade secrets in the process.

    Of course, since MD5 is a very good cryptographic hash function, *any* one-bit change in the source will result in, on average, half of the bits in the result being flipped. So, this method of identifying copied code would only work if the code had never been run through an obfuscator. It would also be defeatable by running the source through a script to have its variable names search-and-replaced with similar names (such as replacing every variable name with a new name consisting of the old name plus "_newname")....

    In short, this might be a useful technique for allowing a third party to look for trivial wholesale copying of code, but it would be useless for finding a motivated miscreant, determined to steal code without being caught.
    • by Trailer Trash (60756) on Tuesday September 09, 2003 @06:56PM (#6916211) Homepage

      So, this method of identifying copied code would only work if the code had never been run through an obfuscator.

      You've hit the nail on the head, possibly without knowing it. The source code needs to be run through an obfuscator *before* shredding. Actually, I'm thinking a special obfuscator, let me explain.

      Let's take a piece of C source, not randomly chosen:

      malloc(mp, size) struct map *mp; { register int a; register struct map *bp; for (bp = mp; bp->m_size; bp++) { if (bp->m_size >= size) { a = bp->m_addr; bp->m_addr =+ size; if ((bp->m_size =- size) == 0) do { bp++; (bp-1)->m_addr = bp->m_addr; } while ((bp-1)->m_size = bp->m_size); return(a); } } return(0); } Now, the structure of the code is 99% of what matters. Variable names can change, but few people would change anything beyond that. Let's modify the code in a couple of important ways. First, all variable names are changed to new names, on a per-line basis. Blank lines and unneeded blanks are all removed. Each statement is on its own line, and formatting styles (such as curly bracket placement) are standardized. malloc(a, b) struct a *b; { register int a; register struct map *b; for (a=b;a->c;a++) { if (a->b>= c) { a=b->c; a->b=+c; if ((a->b=-c)==0) do { a++; (a-1)->b=a->b; } while ((a-1)->b=a->b); return(a); } } return(0); }

      This might not be perfect, but it should do the trick. A programmer can change variable names, spacing, or format, but as long as the code is the same, it'll match. Obviously, changing the code would have an impact, but nearly every line would have to be changed for it to not match, and in a substantial way. That's literally not always possible to even do in a way that would trick this function.

      Anyone want to write it?

      Michael

  • Ups and downs (Score:5, Informative)

    by autocracy (192714) <slashdot2007 AT storyinmemo DOT com> on Tuesday September 09, 2003 @05:07PM (#6915342) Homepage
    Upside: we can maybe help catch more stolen code.
    Downside: Uh... it just came out... and it's making some big, big claims involving fuzzy logic. I think it's gonna need some testing first, eh?

    Also, anybody else think it only works on larger sections of code than just say 10 lines?

  • by YetAnotherName (168064) on Tuesday September 09, 2003 @05:08PM (#6915361) Homepage
    Thanks ESR. You've just put a team of mathematicians at SCO who were somehow related to MIT [mit.edu] out of their jobs.
  • Slim to None (Score:5, Insightful)

    by tomRakewell (412572) on Tuesday September 09, 2003 @05:10PM (#6915382)
    Chances are slim to none that a software company would allow it's "shredded" source code to be publicly released. What happens if the proprietary source is found to violate the GPL?

    Proprietary (closed) source companies have a tremendous advantage over open source software when it comes to violating intellectual property. Who will ever know if they did it? A source code "comparator" eliminates that crucial advantage.
    • Re:Slim to None (Score:5, Interesting)

      by JoeBuck (7947) on Tuesday September 09, 2003 @05:36PM (#6915622) Homepage

      But IBM already has a copy of SCO's code; they licensed it after all. They can release the output of "shred" without violating their agreements with SCO.

    • Re:Slim to None (Score:3, Interesting)

      by k98sven (324383)
      Proprietary (closed) source companies have a tremendous advantage over open source software when it comes to violating intellectual property. Who will ever know if they did it? A source code "comparator" eliminates that crucial advantage

      Not really.. Open-source software usually has a nice setup with mailing-lists, CVS, etc. Most of the code is well accounted-for. The same is not as true with a lot of proprietary software.

      Remember, it's not enough that two pieces of code match to prove an infringement in
  • by zapf (119998) on Tuesday September 09, 2003 @05:11PM (#6915386)
    While I fully support ESR and the rest of the open source movement's defense of Linux against SCO, I have a feeling that this tool's results will not immediately be accepted by established media simply because of ESR's bias. A reporter looking into the SCO story who knows little about open source wouldn't trust a tool made by one side of the disagreement.

    It seems very important to me that "third parties" and experts who are not an integral part of the open-source movement validate that comparator works as intended and is effective at detecting code similarities. Hopefully we'll see some articles on respected sites in the next week or so with conclusive analyses of comparator. Not to mention a chance for someone to use it on SCO's code!

    Oh, and "Yes, I'm being deliberately vague and tantalizing" is quite funny.
    • by Brandybuck (704397) on Tuesday September 09, 2003 @06:26PM (#6916002) Homepage Journal
      A reporter looking into the SCO story who knows little about open source wouldn't trust a tool made by one side of the disagreement.

      Then why would a reporter trust the press releases that SCO puts out on an daily basis?

      The unfortunate reality is that they DO trust them. We may all think this is a joke here in our insular community, but the great majority of reporters report the press releases "as is". Then the analysts come along and refine those press releases into easily digestible chunks. Then the pundits come along with preconceptions based on those chunks. Ever wonder why the SCO stock keeps going up and up and up? It's because the only thing the general public knows about this issue has come from SCO.

      Anything that can help get the truth before the public eye is a Good Thing(tm). A tool that can mathematically "prove" that SCO is lying is valuable, even if most reporters suspect a bias.
  • by TedTschopp (244839) on Tuesday September 09, 2003 @05:12PM (#6915397) Homepage

    This is perhaps a better project and it would be interesting to see this tool run against the source.

    History Flow [ibm.com] The following is from their website:

    history flow
    visualizing dynamic, evolving documents and the interactions of multiple collaborating authors:

    Motivation
    Most documents are the product of continual evolution. An essay may undergo dozens of revisions; source code for a computer program may undergo thousands. And as online collaboration becomes increasingly common, we see more and more ever-evolving group-authored texts. This site is a preliminary report on a simple visual technique, history flow, that provides a clear view of complex records of contributions and collaboration.

  • by Paradox (13555) on Tuesday September 09, 2003 @05:12PM (#6915399) Homepage Journal
    Well, I was looking at ESR's description of the code (I haven't read the code yet), and it seems to say that he takes 3 line slices, MD5s them, then compares them for identical points. I'm sure he compensates for funky whitespace and whatnot like diff and patch do...

    But if even one bit of the source is different, the MD5 hash will be quite different. So, the code slices have to be IDENTICAL. This is not a very good system because a simple find-replace could defeat it. A variable's name changed by one letter, or even capitalization, will defeat it.

    Unless the code reveals much more complex tricks than ESR describes in the help file, this tool wouldn't be much use in the SCO case. Hell, it wouldn't be much use catching college class cheaters even.

    • But SCO claims that a lot of code has been copied *as-is* along with the comments.

      The tool ought to be able to highlight all those flagrant cases (if any) and the report generator would then generate something that would be perused by a human.

    • except that according to SCO millions of lines were copied VERBATIM into linux.

      Verbatim would give a matching md5 sum, sysv code isn't tough to get your hands on (especially since IBM has it, as well as their own code they supposedly contributed). Making the md5 hashes will be a breeze.
  • by Anonymous Coward on Tuesday September 09, 2003 @05:15PM (#6915435)
    check out this research project coming out of berkeley CAP [berkeley.edu]

    Drop in the code you are interested in and it will tell you where its found in a bunch of open source stuff, including the linux kernel.
  • by expro (597113) on Tuesday September 09, 2003 @05:18PM (#6915458)

    Think of the chance that any given line of source code in an arbitrary program is repeated somewhere else in a large open source program such as the Linux Kernel. This is even more true if some degree of fuzziness is added to handle changes such as adding or removing spaces in insignificant places, removing comments, (and there are many other things like brace style which affect multiple lines so you might want to physically reformat between lines to a standard format....

    If the number of lines is even only 1% that are found somewhere in the open source code base, I think a source who wants to keep their code base secret will have a big problem with someone computing the checksums. In reality, I wouldn't be suprised to see a much-higher percentage of lines leaked this way. And this is not the only way leaking can occur (think of application of simple cryptography).

    I would not want to be the one publishing the checksums of the closed source due to possible legal liability. The checksums are a derived work in any case.

  • by pclminion (145572) on Tuesday September 09, 2003 @05:30PM (#6915566)
    int main()
    {
    printf("These source trees appear to be entirely different!\n");
    return 0;
    }
    • by greppling (601175) on Tuesday September 09, 2003 @06:32PM (#6916040)
      ...how to write good user interfaces. With coders like you we will never achieve complete world domination. The correct program is, of course, s.th. like this:

      int main()
      {
      int i;
      printf("Comparing source trees...\n");
      sleep(2);
      printf("Check started.\n");
      for (i = 1000; i--;) {
      printf(".");
      sleep(1);
      if (i % 100 == 0)
      printf("\n%d0 percent remaining\n", i / 100);
      }
      printf("\n\nThese source trees appear to be entirely different!\n");
      return 0;
      }
  • by The Lynxpro (657990) <lynxpro@noSPAM.gmail.com> on Tuesday September 09, 2003 @05:42PM (#6915669)
    Have any of those techno-Rabbis run a comparison search with their "Bible Code" program on SCO? Did it come up with the phrases "bankrupt in 2004," "full of camel dung," and "Serpent of Utah"? How about running the "Bible Code" on Unix System V. code? Considering SCO's fondness for converting code over to Greek symbols for their presentations, converting to sanskrit, Hebrew or Aramaic shouldn't be a problem...
  • by klui (457783) on Tuesday September 09, 2003 @05:59PM (#6915786)
    It will just tell someone two trees are similar/identical. The important thing to prove in court is who copied from whom.
  • Open Source (Score:4, Interesting)

    by digidave (259925) on Tuesday September 09, 2003 @06:46PM (#6916143)
    THIS is exactly why Open Source works. It's not because of IBM or Red Hat or geeks from Finland. It's because people in the community are willing to step up to any challenge.

    Thanks, ESR.
  • by LinuxParanoid (64467) * on Tuesday September 09, 2003 @06:47PM (#6916149) Homepage Journal
    Pardon me, but a lot of you guys are missing the point of this comparator.

    1) There are people out there with legit source licenses to SVR5 source trees. And not just Unix OEMs. Various people in large companies with SVR4/5 source licenses etc.

    2) Such people cannot release the source code, and may (if paranoid of how they interpret 'derived works') not want to publish hashed MD5 codes of SVR5.

    3) However, it should be possible for a legit SVR5 source licensee to publish openly a list of areas of code that are similar across trees, without that list being either A) a derived work, B) violating their NDAs (um, do check the fine print first though) and C) spending tons of their own, presumably expensive time, digging through stuff.

    Then Linux advocates can then sift through the resulting large pieces of code and doublecheck/crosscheck the origins of it. At the very least, we'd have a public list of suspicious areas of Linux and could determine that certain parts are A) BSD-licensed, B) are verified as original by a known Linux coder, and C) don't fall into the above categories and remain 'suspicious'. This presumably is what ESR is referring to by "various persons will apply it in useful ways. Yes, I'm being deliberately vague and tantalizing". Let's say that its likely the percentages in A and B will be large.

    Of course it's true that there could be code that this primitive tool doesn't catch. But SCO probably started their analysis by using tools like this also. Looking through millions lines of code by hand is no cakewalk, so one will inevitably start with code like this in such an investigation. (Unless one is concerned about one specific predetermined critical/sensitive piece of code.)

    Oh, and the other thing about this tool that is nice IMHO? It demonstrates a "good faith" effort on the part of Linux advocates and coders to correct the problem -- despite the barriers raised by SCO (no code release except via NDA).

    Finally, running this tool across a Linux and a BSD release should turn up some data that is both interesting and relevant for this dispute. I'm almost tempted to try that myself.

    --LP
  • by Ivan the Terrible (115742) <vladimir AT acm DOT org> on Tuesday September 09, 2003 @07:58PM (#6916576) Homepage

    I'm interested in algorithms that could be used to compare code. Moss and CAP from Berkeley are not interesting because the algorithm is secret (AFAIK).

    What algorithms other than ESR's comparator are there? (I recall but can't locate a recent comment on Slashdot that said something like "most plagiarism detection programs used by professors use the XXXX algorithm".)

  • by Dr. Smeegee (41653) * on Tuesday September 09, 2003 @08:46PM (#6916908) Homepage Journal
    He developed a Callcenter Training Utility for our company in the early 80's. It used genetic algorithms to generate simulated customer complaints that were _very_ realistic, even to the point of using sample voices to "whine". Of course, the helpdesk trainees hated it...

    But hey, the mewling was featureful.
  • by Mostly a lurker (634878) on Tuesday September 09, 2003 @09:26PM (#6917226)
    As currently designed, Shred would obviously not defeat deliberate source misappropriation. If (big if) the method could adapted such that it could not be easily fooled by a determined violator (and without revealing how the code works) then I believe registration of the results should be required by law. BUT ...

    In order that the method should not be fooled by simple changes, at least the following is required

    * White space must be ignored

    * Comparison must be at the statement level, not the code line level

    * Variable names must be replaced by standard placeholders

    * Routine names, other than standard library calls, must be replaced by standard placeholders

    * (Probably difficult) logic will be needed in the tool to detect and ignore noops: how do you deal with

    i++;

    %include noop.i;
    a[i]=b[i];

    The trouble is: a high proportion of the code sections thus simplified will fall into a relatively small number of possibilities, vulnerable to dictionary type attacks. Thus, most of the code could be reconstructed, though admittedly as obfuscated source code. IMHO this provides a valid objection to its use.

  • Doesn't even compile (Score:3, Informative)

    by tvm662 (232083) on Tuesday September 09, 2003 @09:50PM (#6917473)
    Has anyone else tried to compile Eric's code?

    >gcc --version
    2.95.3

    >make /usr/bin/gcc -c -g main.c
    main.c: In function `report_time':
    main.c:311: parse error before `int'
    main.c:312: parse error before `int'
    main.c:316: `buf' undeclared (first use in this function)
    main.c:316: (Each undeclared identifier is reported only once
    main.c:316: for each function it appears in.)
    main.c:317: `minutes' undeclared (first use in this function)
    main.c:317: `seconds' undeclared (first use in this function)
    make: *** [main.o] Error 1

    Looks like Eric has been coding too much c++ or something. I'm not a c coder myself, so I might be wrong, but don't you have to declare all the variables in a block of c code before using them. In report_time, he doesn't seem to have followed that rule. Maybe he might check his code on a number of compilers before declaring he has "perfected it".

    Eric here's my patch:

    --- main.c 2003-09-10 00:28:37.000000000 -0300
    +++ main.c.fixed 2003-09-10 00:29:55.000000000 -0300
    @@ -306,12 +306,17 @@

    if (mark_time)
    {
    - int elapsed = endtime - mark_time;
    - int hours = elapsed/3600; elapsed %= 3600;
    - int minutes = elapsed/60; elapsed %= 60;
    - int seconds = elapsed;
    + int elapsed;
    + int hours;
    + int minutes;
    + int seconds;
    char buf[BUFSIZ];

    + elapsed = endtime - mark_time;
    + hours = elapsed/3600; elapsed %= 3600;
    + minutes = elapsed/60; elapsed %= 60;
    + seconds = elapsed;
    +
    va_start(ap, legend);
    vsprintf(buf, legend, ap);
    fprintf(stderr, "%% %s: %dh %dm %ds\n", buf, hours, minutes, seconds);
  • Useful tool (Score:3, Insightful)

    by Julian Morrison (5575) on Tuesday September 09, 2003 @09:50PM (#6917475)
    I can see this tool becoming helpful for so much more than smashing SCO. Any situation where data comparison is useful, but the data itself must remain secret. All paranoid types (corporate or governmental) will love it. Lawyers could make much use of it.

    And, given the dataset it generates, it could be extended to do other useful things such as detect redundant or cut-'n-pasted code, including bugs of the "pasted it in twice" sort.

Any sufficiently advanced technology is indistinguishable from a rigged demo.

Working...