New Method To Detect and Prove GPL Violations

qwerty writes "A paper to be presented at the upcoming academic conference Automated Software Engineering describes a new method to detect code theft and could be used to detect GPL violations in particular. While the co-called birthmarking method is demonstrated for Java, it is general enough to work for other languages as well. The API Benchmark observes the interaction between an application and (dynamic) libraries that are part of the runtime system. This captures the observable behavior of the program and cannot be easily foiled using code obfuscation techniques, as shown in the paper (PDF). Once such a birthmark is captured, it can be searched for in other programs. By capturing the birthmarks from popular open-source frameworks, GPL-violating applications could be identified."

new use of old trick

I used to be a research assistent, and at university, we used this technique to see if students copied their assignments. They could rename variables, move pieces of text, change comments all the way they liked, but the execution profile stayed the same. We caught a lot of students, and they never figured out how we did it.

Re:new use of old trick

How did you know they were cheating and didn't derive their similar approaches from a common origin (presumably material that was presented in class or else from the textbook)? My experience with marking for a computer science professor showed that about 80% of the students approached any given programming assignment almost exactly the same way in terms of their final implementation... their common origin being something the teacher described during a lecture.

Re:new use of old trick

How did you know they were cheating and didn't derive their similar approaches from a common origin (presumably material that was presented in class or else from the textbook)?

Amen to that. This is an old story, but I think it bears repeating. A friend of mine and I got "caught" turning in identical code for an assignment. I mean, identical. Same structures, variables, types, layout - everything. However, we wrote our programs separately and never saw each others' until our teacher asked about it.

It sounds improbable, but consider that:

1. We both directly transcribed variable names from the homework assignment. A sentence like "it is a fatal error condition for the user to specify a negative number of tasks" became "assert(numtasks >= 0);".
2. We used the same editor and the same indenting style.
3. We had done much of our homework together in previous classes because we tended to take the same approach to solving problems.
4. The assignment wasn't terribly complex to begin with, so the resulting code was only a few pages long.

We had a teacher who trusted us and we were both good students with good test grades, so it was dismissed as a humorous coincidence. I'm glad a human was willing to listen to our explanation and not just go along with the findings of an automated tester.

Re:new use of old trick

I take it your code was flawless?

Of course! ;-)

people who write flawless code can easily prove their innocence by answering a couple of questions about the implementation on the spot.

I think there was a bit of that, too: (pointing at me) "why did you do this?" "Because of this requirement in the last paragraph." (Pointing at friend) "and why didn't you use this approach?" "That wouldn 't have worked because of this part here."

Re:

just to demonstrate that this sort of overlap isn't just CS undergrads doing homework assignments, take a look at Ken Thompson's Turing award lecture [bell-labs.com], particularly this section:

In the ten years that [Dennis Ritchie and I] have worked together, I can recall

Re:

Or you could just use MOSS http://theory.stanford.edu/~aiken/moss/ [stanford.edu] (or other) like everyone else.

Re:

Well, for the class I TA'ed, it was probably available, just not widely popular yet. Of course, cheaters are usually easy to catch, so even simple systems work pretty well. So, in their attempt to save time and effort, cheaters are often are bad at cover

Re:

Not really... That argument assumes that all classes teach the same subject: compromising verification systems and covering your tracks. Most classes in CS teach something else, with some examples being computer graphics, machine vision, and game program

No, really

lets just set the code free. lets not chase it down the street to make sure it stays free, just let it go as it will.

Re:

That was akin to my first thought: If opensource code is really so superior to closed source code, and if the world would be better off if all apps had been built from those codebases, then shouldn't we *encourage* it to be "pirated", for everyone's net b

Re:

You can use the BSD license for your code if you unconditionally believe that "more copies of good code = better world". Heck, in many countries you can put code directly in the public domain. For those who think that authors of good (open) code need to be

Re:

That was akin to my first thought: If opensource code is really so superior to closed source code, and if the world would be better off if all apps had been built from those codebases, then shouldn't we *encourage* it to be "pirated", for everyone's net be

Re:No, really

For Open Source code, you are right. The Open Source movement believes in the superiority of the 'bazaar' development mode. If you try to create a closed fork then you are going to fall behind the open version, and have to spend a lot of time and effort merging changes from the main tree.

The Free Software movement, however, believes that code which protects the user's freedoms to use, modify and distribute it is intrinsically superior, and that people who wish to write code that does not respect these freedoms should not be aided by being able to use the work of those who do.

As such, an Open Source advocate would not mind, because the closed copy would quickly become inferior. A Free Software advocate would object, because their work would be being used for (in their view) unethical purposes (denying end users their freedoms).

Re:

If someone wants to invest time and effort into a closed fork, good on them. Everyone else still has access to the original branch, and the creator of the fork isn't messing with any of those rights. I don't see any imbalance, the authors are not falling v

Re:

I have released code under BSD license (as well as GPL, ZLib/LibPNG/, Boost, public domain and proprietary, and probably a few others).

The LEAST of my concern in releasing ANY open source is some childish popularity contest.

The only valid reason for me has

Re:No, really

That won't do. The GPL is really more of a social instrument than a software license, so for people like Stallman a BSD-style license (which is just one step above public domain and true freedom) would be unacceptable. A lot of bandwidth and keyboard lubricant has been spent over the years to ensure that everyone thinks the GPL is the "best" software license - and the thousands of developers that buy into the FSF "freedom, with caveats" spiel by using the GPL (because well, that's what everyone uses) without really understanding what it's for are part of that problem.

As you can imagine I really don't like the GPL or the FSF or Richard Stallman or any of his friends too much. While I recognize their contributions I think that they've fallen into the trap of trying to force everyone to convert to what has become a quasi-religion where the Inquisition is more important than celebrating mass.

Re:

keyboard lubricant

I've never heard it called that before.

Re:

Making the code freer than the GPL lets eg. Microsoft's embrace, extend, extinguish a whole lot easier. Now they just have to copy/paste and slightly modify the code, compile it, and pass it off as theirs. Some of us don't like that.

Re:

I consider BSD to be a superior server environment to Linux, and so far it's doing quite well.

Re:

MS has 70,000 employees, most of which are mediocre. In fact, that's almost the essence of their development model --- throw thousands of crappy developers at a problem and excrete a solution that is just workable enough to make some money.

Re:No, really

You know, I'm absolutely tired of the BSD trolls that claim that the BSD license is "freer", not because I have a beef with the BSD, simply because your definition of "freedom" is ludicrous.

There are no absolute freedoms. Freedom to infringe on other's rights or freedoms gives more freedom to yourself, but limits it to other members of society. So long as there are things that cannot be owned or achieved communaly without side effects to others, freedoms have a limit, that is the actions that you cannot do so that others can do them.

The GPL definition of freedom is that a sofware and derivatives must always, under all conditions, be free. Yes, it a restriction to the developer who would wish to close up his source and use a GPLed piece of code, but it is an additional freedom to all the users who now have access to this source, which would have otherwise been denied.

Analogy time: the King is free to treat his peasants as dogs if he wished and if he has sufficient power to repress any opinions the peasants would have about that. The peasants, however, are limited by the freedoms the king has. Therefore the balance of freedoms for a more equal society would be that the king's freedoms be limited in order to allow the peasants to live their life.

So as you said, the GPL is also a social instrument, but it is no less free than the BSD; it simply distributes freedoms in a different matter. If you have a problem with that, use whichever license you wish to use. But don't go around accusing the GPL is limiting freedoms when it gives others freedoms that the BSD could never guarantee.

Re:No, really

You know, I'm absolutely tired of the BSD trolls

If by that you mean "you have a different definition of what freedom is, therefore I don't like you" then sure, I'm a "BSD troll" or whatever.

your definition of "freedom" is ludicrous.

GPL -> Distribution restrictions.
BSD -> No restrictions.
No restrictions -> More freedom.
More freedom -> Possible unsavory side effects that people choose to live with

Isn't logic great?

The GPL definition of freedom is that a sofware and derivatives must always, under all conditions, be free.

BSD has a similar one, except that it doesn't place restrictions on how that happens. No one can make BSD-licensed software "non free", it will always be available to everyone. The only difference is that it might not benefit from coerced third party improvements, but that's what you sign up for.

it simply distributes freedoms in a different matter

The Kool-Aid is strong with this one.

But don't go around accusing the GPL is limiting freedoms when it gives others freedoms that the BSD could never guarantee.

BSD licenses guarantee absolutely nothing. Here's the code, do whatever the heck you want with it. The perceived benefits to using the GPL are nice, but please don't insult people's intelligence by claiming they result in more freedom. A restriction to ensure X or Y is still that - a restriction. The distribution restrictions on the GPL are designed to further Stallman's social causes (some of which I actually agree with). If you feel that's fine, then by all means use the GPL. That's your choice.

Re:No, really

GPL -> Distribution restrictions. BSD -> No restrictions. No restrictions -> More freedom. More freedom -> Possible unsavory side effects that people choose to live with

GPL -> Code will always be open and derivatives will stay that way
BSD -> Code can be closed off and new improvements to it can remain closed off forever.
Always open code -> More freedom
Sometimes open code -> Permanent loss of freedom with regards to that code.
Indeed, logic is great.

BSD has a similar one, except that it doesn't place restrictions on how that happens. No one can make BSD-licensed software "non free", it will always be available to everyone. The only difference is that it might not benefit from coerced third party improvements, but that's what you sign up for.

I never said that you can't sign up for that if so you wish, but code is always used within contexts, and when used in the context of proprietary software, any improvements on the code will be lost, any bug fixes will be lost, any added functionality will be lost.

Sure, some people will build upon it, but losing the obligation of putting the improvements back into the codebase means that it will eventually stagnate, and that the improvements that could have been used for the good of everyone who contributed can be denied at will. Look at FreeBSD with OS X: Apple got the foundation of their OS for free, and after that they simply closed up the rest at will. Perhaps the Apple folks got to improve their memory management, or add some new DRM techniques. Whatever they've done, the FreeBSD devs will never get to see it.

If they don't mind as users and developers to see their work used to create a proprietary, vendor-locked platform then it's their prerogative; as a used and dev I prefer to make sure that my code is an established base of constant improvement. With the GPL they're empowered and free to do that; with BSD new parties are empowered to do whatever and completely ignore original creators aside from the required attributions.

Notice that I'm not saying the BSD license is more free; it is equally free, but shifting freedom to new developers and vendors to be,IMO, lazy bastards and profiting for nothing, while GPL shifts it to original developers, contributors and users to get reciprocal treatment from others. You're free to think that the former is more important; I belive the latter brings greater benefits to everyone in the long term.

BSD has a similar one, except that it doesn't place restrictions on how that happens. No one can make BSD-licensed software "non free", it will always be available to everyone. The only difference is that it might not benefit from coerced third party improvements, but that's what you sign up for.

No one is coercing anyone here. If you had read and understoof the GPL, and it looks like you haven't, you'd know that the conditions apply only to those who want to redistribute software. If you want to keep your patches to yourself you can do that and it's your right, but if you're going to be using other's code to sell it or gain from it you have to abide by the creator's conditions. Going back to my point about freedom, perhaps as distributor you have less leeway regarding your changes, but your users have just gained the guarantee that they'll always be able to see and change the code. The BSD could not have done that.

BSD licenses guarantee absolutely nothing. Here's the code, do whatever the heck you want with it. The perceived benefits to using the GPL are nice, but please don't insult people's intelligence by claiming they result in more freedom. A restriction to ensure X or Y is still that - a restriction. The distribution restrictions on the GPL are designed to further Stallman's social causes (some of which I actually agree with). If you feel that's fine, then by all means use the GPL. That's your choice.

You hit the nail on the head. Th

Re:

Oh I think everybody understands it just fine because it's basicly "Modify it any way you want. If you distribute it, source code goes with it". Ok so it's not free as in public domain, but who really has a problem with the GPL? Only those that want to tak

FSF and Stallman have endorsed permissive licenses

The GPL is really more of a social instrument than a software license, so for people like Stallman a BSD-style license (which is just one step above public domain and true freedom) would be unacceptable.

Not so fast. The GPL FAQ [gnu.org] states that there exist situations where a permissive license is appropriate, in particular short programs [gnu.org] and web site templates [gnu.org]. Mr. Stallman has also endorsed the use of a permissive license for a library designed as the refere [xiph.org]

Re:

The GPL vs BSD "freedom" argument is really boring semantics. Whether the GPL is freedom, slavery, communism or whatever else you want to call it is irrelevant to me: It does precisely what I want, which is why I use it.

Re:

The code doesn't need freedom. People need freedom. Let the bad guys incorporate GPLed stuff and they are likely to become an issue because they'll enhance it and defend it as it were all their own, against similar enhancements done to the GPLed branch.

Bes

A couple of things....

What is the false positive rate for this method? What if two programs just happen to do the same thing and the authors happened to choose similar ways to do it. Would this method conclude that one originated with the other? It's not a copyright violation because neither is a derivative work of the other.

Also, it occurs to me that this method would probably not be as useful as expected for detecting GPL violations. It would think it would only be effective for checking where you have source code available, or at the very least enough symbol table information to make comparisons, which you are not likely to have if somebody is violating the GPL because that implies no source code anyways (and almost certainly no symbol table information for the binary).

Re:

More to the point what is the false negative rate? There is tons of really useful code out there that doesn't make any system or library calls at all. It just takes data, processes it in some way, and hands back the results. That description could apply to

Re:A couple of things....

My guess is that it would work much better for java and possibly C++ than more concise languages which don't have tonnes of implicit calls and inheritances. And even with OO languages like java, I'd think that simply adding a try in the middle would change the fingerprint quite a bit.
Also worth considering is what a compiler optimiser might do -- they can be quite good at rearranging code different ways depending on whether optimising for speed or code size, and what the target is. That's probably another reason why this might work better with java, which only has rather rudimentary jit optimiser.

If this tool can help identify some infringing code, that's well and good, but I wouldn't rely on it, wouldn't think it would add much if any legal weight, and neither would I think it could replace a thousand eyes.

Anyhow, the real problem, as I see it, with identifying open source code pilfered and added to a closed source project is that you generally aren't allowed to reverse engineer the code itself to see what it actually does. So even if you're Very Damn Sure that a piece of commercial software illegally uses open source and sells it as its own closed source, you're not allowed to investigate and come up with evidence. You'll have to file a suit and get a judge to order the code examined, and with only a good hunch to go on, and no way to document a financial loss, and probably not having too deep pockets yourself, that's rather unlikely to go anywhere.
Which is why I think it's important that we support institutions like FSF, which can occasionally fight the battle on behalf of the little guy.

Regards,
--
*Art

Re:

Anyhow, the real problem, as I see it, with identifying open source code pilfered and added to a closed source project is that you generally aren't allowed to reverse engineer the code itself to see what it actually does.

The number of cases where this is a

Re:

There is tons of really useful code out there that doesn't make any system or library calls at all. It just takes data, processes it in some way, and hands back the results

Are you sure? You know that read and write are system calls? And that printf, sqrt, exp, etc are all library functions? Even trivial code makes a lot of system calls. A hello world program, in C, on Linux, makes 27 system calls (number from strace).

Re:

probable cause? if two programs execute in virtually identical ways there is a reason to investigate. it doesnt catch them all but it is better than nothing.

Clean room could replicate signature.

An identical library call signature for a nontrivial part of the execution could be produced by a clean-room analysis or even independent development of an equivalent component. Neither of these is a GPL violation.

This is not to say that the technique wouldn't be useful for hunting down GPL violations. But a positive is not difinitive by itself.

Meanwhile code obfuscation (even automatically generated obfuscation) could easily modify at least the timing, if not the order, of such calls.

Nevertheless this is a powerful tool: An hunk of GPL code that hasn't had its flow obfuscated systematically (even code that HAS been obfuscated but not systematically) will have large swaths of code that trips the detector. And it doesn't require reverse engineering until after the alarm goes off.

Good job, guys.

Re:

Meanwhile code obfuscation (even automatically generated obfuscation) could easily modify at least the timing, if not the order, of such calls.

(Yes I know that the article says it can't. But that refers to the usual sort, which is directed at hiding the s

Coming soon...

GGA! The GNU Genuine Advantage program!

Sweet Mother of All Revolutions

Pitchfork? ... Check
Torch? ... Check
Map of Corporate Castle locations? ... Check
FSF Lawyers programmed to be speed dialed in emergencies? ... Check
Desire to burn the non-believers? ... Check

Okay, I'm ready! What IRC Channel are we meeting in?

Other languages

I looked through the paper, and it is cool stuff. But I couldn't see where it supposed the system would work well for other languages, and I wonder if it really would be so good.

Java has a very large standard library that is always dynamically linked, and hence can easily be instrumented as the technique requires. C allows static linking which would make such hooking much more difficult. Additionally Java executes in a very standard environment due to the Virtual Machine, where as other languages may have varying ABIs type sizes and other properties that could add significant noise to the birthmark.

That said, system calls are always hookable and reasonably standard, so maybe this technique could be applied successfully there for malware detection or similar?

Heh..

When people go to these lengths to prove misuse of commercial licenses, they're called fascists. When it's done to prove misuse of free licenses, it's OK.

I see the community is still working as it always has.

Re:

It may be news to you but non-commercial licenses are AFAIK universally considered non-free (where as you see to imply the two are mutually exclusive). And when has anyone ever had any problem with people going to lengths (whatever that means) to prove li

Very Cool

This is very cool and potentially useful. By itself, it wouldn't be enough to force compliance or win a violation suit, it could well be enough to meet the threshold for filing a suit and forcing source code analysis in discovery. Really, it is a great t

Just great

Instead of coding open source projects, now we're coding projects to detect license violations.

Next, the Open Source Business Software Alliance and raids by the Secret Service...

When is the last time we read anything about open source that wasn't about lic

Re:

Instead of coding open source projects, now we're coding projects to detect license violations.

Well one person has as part of some academic research. You see, the beauty of FLOSS development is everyone can code what they enjoy coding, and that you don't have to help anyone but can instead do something you prefer.

I won't even bother addressing y

Wine?

How well does it work with the Wine versus Windows comparison?

It's not theft

a new method to detect code theft

I realise this is going off on a tangent, but I'm concerned about the use of the word theft. Usually I'm one of the first people to jump up and down when I hear the RIAA or MPAA accuse people of stealing, and I've notice

Read comments and think MS patent claims on linux.

False positives....

The story is presented with a stage light focused on linux but then the house lights come up and show linux in jail along with most of the audience.

This is just one paper for one Automated Software Engineering (ASE) conference.

But if you

Re:

Congratulations, in something that is going to be a GPL vs. BSD flamewar you are starting a vi vs. Emacs flamewar.

I'm really touched.

Re:

i actually figured out what the hell GPL is.

Interesting. Up to now I only knew about the GNU GPL and the Affero GPL. I guess the hell GPL differs from the GNU GPL that you have not only to give away your source, but also your soul?

Re:And who the fuck might I ask is to spend the ti

Good thing no one asked you to. GPL code is open-source, so "keeping it out of their grubby little hands" is not an option or even wanted. You had probably better come to understand the purpose of the GPL and what a GPL violation is before you post.

Re:

The use case for this is that you release a piece of code, and then find someone else releasing code that does something similar. You run this program on your code and theirs, and there are three likely outcomes:

• They didn't copy your code, and the progr