Competition Produces Vandalism Detection For Wikis 62
marpot writes "Recently, the 1st International Competition on Wikipedia Vandalism Detection (PDF) finished: 9 groups (5 from the USA, 1 affiliated with Google) tried their best in detecting all vandalism cases from a large-scale evaluation corpus. The winning approach (PDF) detects 20% of all vandalism cases without misclassifying regular edits; moreover, it can be adjusted to detect 95% of the vandalism edits while misclassifying only 30% of all regular edits. Thus, by applying both settings, manual double-checking would only be required on 34% of all edits. Nothing is known, yet, whether the rule-based bots on Wikipedia can compete with this machine learning-based strategy. Anyway, there is still a lot potential for improvements since the top 2 detectors use entirely different detection paradigms: the first analyzes an edit's content, whereas the second (PDF) analyzes an edit's context using WikiTrust."
First Vandal (Score:1, Interesting)
here we are knocking at the friendly gates of wikipedia and now they want to throw us out.
Is there no hospitality in this world ?
ad: Searching for experienced pillagers and fortress busters. Exp 5 years min. Trebuchet skills a plus. Bring your own broadsword.
20% with no false positives? (Score:4, Insightful)
If the algorithm can detect 20% with perfection then that must constitute extremely low hanging fruit. That type of vandalism is just annoyance. It is so obvious that the end user readily recognizes it as such and can skip over it or revert the edit.
The real issue is disinformation, which is vastly more subtle. The only defense is fact-checking or seeking out references. If the algorithm is capable of recognizing that kind of vandalism then the developers should have the software writing all the articles in the first place, because it'd have to be pretty spectacular to manage that.
Re: (Score:2)
The major problem now is that 99% of all good edits submitted to wikipedia are reverted anyways as false positives.
The reason for this is that corrupt administrators do nothing to stop it, and corrupt idiots wanting to become admins just sit all day on the semi-automated tools like "Twinkle" or "Huggle" reverting anything in sight to get their edit counts up.
The real issue is disinformation, which is vastly more subtle. The only defense is fact-checking or seeking out references.
While true, the larger prob
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
Care to show us even one article where 99% of good edits are reverted? Remember, that will mean that over 99% of all edits are reverted.
not if there are bad edits that are not reverted.
Re: (Score:2)
Re: (Score:2)
Depends on your definition of bad edits. Take for example an edit with useful information but unsourced or poorly worded. Delete or keep and try to improve?
Increasingly WP seems to be going for the former. Ditto with useful articles that are about niche subjects, particularly software, which are deemed not notable enough to keep. Whatever happened to "Wikipedia is not paper"?
Re: (Score:2)
the definition of good or bad edits has nothing to do with my previous comment.
if 50% of a page edits are good edits, and the other 50% are bad edits, if none of the bad edits were ever reverted, 99% of good edits were reverted, only 49.5% of all edits were reverted.
It's just mathematics. nothing to do with WP politics of edition and reversions.
On a side note, I agree with you about niche subjects. The notability threshold should be lower.
Re: (Score:2)
I wasn't really arguing with your mathematics, just pointing out that any kind of measurement of good/bad edits is subjective.
Re: (Score:2)
The major problem now is that 99% of all good edits submitted to wikipedia are reverted anyways as false positives.
That's just nonsense, and you know it. It does indeed happen that good edits are reverted, but it does not happen 99% of the time, not even in close to that.
It's hard to say what the bigger problem is -- good edits that are reverted -- or bad edits that aren't. My guess is that both these problems are about equal at the moment, and neither of them are particularily large. That is, Wikipedia s
Re: (Score:2)
If the algorithm can detect 20% with perfection then that must constitute extremely low hanging fruit. That type of vandalism is just annoyance. It is so obvious that the end user readily recognizes it as such and can skip over it or revert the edit.
You have to consider that the people doing the vast majority of vandalism reversions aren't the end users, it's registered wikipedians who maintain articles as a hobby. Automatically reverting 20% of the vandalism means contributors have that much more time to spend verifying uncited claims in other articles.
Re: (Score:2)
Just run the thing a few times and you'll get almost all of it, duh!
20%? (Score:2)
Also, what about more subtle vandal
and the reversionists? (Score:2, Insightful)
The people who "own" a page with the assistance of powerful insiders and revert any changes to their "pet" pages, even spelling fixes or simple corrections to bad information?
Will edits of *those* insiders, who are ruining wikipedia for the rest of us, be flagged by the algorithm as vandalism?
Re: (Score:3)
Re: (Score:2)
Can you show me where the goal of wikipedia is documented as being so low that only spelling fixes and simple corrections are needed? That sounds more like recaptcha than a wiki.
Re: (Score:2)
Re: (Score:2)
Can you show us a page where any changes, even spelling fixes or simple corrections, are reverted?
No, except in edit wars, I haven't seen spelling fixes randomly reverted.
But I have seen pages where simple factual errors have been corrected, along with citations, AND even a note about the edit on the Talk Page, and they are still reverted. It most often happens on articles too obscure to be policed well that are likely to attract people with agendas. (For example, I've seen both left-wing and right-wing religious crazies peddling their incorrect historical/factual assertions on obscure pages on reli
Re: (Score:2)
[Citation needed]?
Which article? Which admin? When?
Wikiality (Score:2)
100% effective method (Score:4, Funny)
Or, you know, just keep applying the first setting that always correctly detects 20% of vandalism on the 80% that's left over, until there's nothing left. Problem solved.
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
In fact, can you continue clarifying 20% of the 80% left from each previous clarification until you reach infinity? I think that'll do it.
Re:100% effective method (Score:4, Funny)
I suppose I should now go and vandalise the article to keep in the spirit of things. Hang on, I'm half way there...
Re: (Score:2)
Re: (Score:1)
Manual double checking? (Score:2, Interesting)
Re: (Score:1, Informative)
According to the 2nd link, the vandalism rate on Wikipedia is 2391/28468 = 0.084, not 0.60!
The second link actually says:
The corpus compiles 32452 edits on 28468 Wikipedia articles, among which 2391 vandalism edits have been identified.
So that is a vandalism rate of 2391/32452 = 0.074. When I do the math I get 33% of all edits requiring a manual check. The vast majority of them are false positives.
0.074 * (0.95-0.20) + (1-0.074) * 0.30 = 0.0555 + 0.2778 = 0.3333
Re: (Score:1)
i hope they include this algorithm asap... (Score:2)
...because I am tired of my small edits here and there to be classified automatically as vandalism.
There is a pretty simple heuristic (Score:3, Interesting)
This comes from personally maintaining some 200+ wikis on Wikidot.com.
There are two kinds of vandals: those in the community of contributors, and those outside it. The first class of vandals cannot easily be detected automatically but when a wiki is actively built, the community will easily and happily fix damage done by these. The second class are usually spammers and come along when the wiki is stale. They are easily detected by the fact that a long static page is suddenly edited by an unknown person. It's very rare to find a real edit happening late after a wiki has solidified. We handle the second type of vandalism trivially by getting email notifications on any edits.
Trick is, wikis (maybe not Wikipedia but then certainly individual pages) don't have random life cycles but go through growth and stasis.
Re: (Score:2)
The second class are usually spammers and come along when the wiki is stale. They are easily detected by the fact that a long static page is suddenly edited by an unknown person. It's very rare to find a real edit happening late after a wiki has solidified.
Ah... now I know why people revert my generally anonymous but high quality edits on neglected articles. Anyone who edits a dormant article must be a spammer or vandal? I don't think this is true.
Trick is, wikis (maybe not Wikipedia but then certainly individual pages) don't have random life cycles but go through growth and stasis.
While I guess you're correct in general, I've seen quite a few situations on Wikipedia where a new user coming in and taking a look at an established article actually leads to a period of revision, reconsideration, and perhaps growth on a given page.
I'm not saying your opinion isn't correct, but it is an overgen
top 2 (Score:3, Insightful)
This implies that the lower-scoring detectors are less valuable in terms of looking for sources of improvement. That's not true, and that wasn't stated in the paper's "Conclusions" section. If the lowest scoring detector finds 5% of the bad data, and it's a different slice from what the other detectors find, then that's quite valuable.
Machine learning - right (Score:5, Informative)
Wikipedia already has programs which detect most of the blatant vandalism. Page blanking and big deletions are caught immediately. Deletions that delete references generate warnings. Incoming text that duplicates other content on the Web is caught. That gets rid of most of the blatant vandalism. It's not a serious problem on Wikipedia.
The current headaches are mostly advertising, fancruft, and pushing of some political point of view. That's hard to deal with using what is, after all, a rather dumb machine learning algorithm that has no model of the content or subject matter.
There already IS a competitive angle (Score:3, Insightful)
They already compete to be the first to revert edits they disagree with.
Mine works 100% of the time. (Score:2)
Hah, bout time. (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
And you want to see how many other pages get taken down in the hunt for the fake.
Good troll, bro :).
Rules can only get so much (Score:3, Informative)
Re: (Score:1, Informative)
It looks like the winning entry [uni-weimar.de] uses all of those attributes plus a bunch more. From pages 3-4 of the paper.
Vandals are likely to be anonymous. This feature is used in a way or another in
most antivandalism working bots such as ClueBot and AVBOT. In the PAN-WVC-
10 training set (Potthast, 2010) anonymous edits represent 29% of the regular edits
and 87% of vandalism edits.
Long comments might
Non-AI algo that can tell good edits from bad? No. (Score:2)
I suspect that, if the regular edits weren't misclassified by the algorithms then:
a) the reference classification was incorrect itself
b) they were much too convenient samples, compared to the kind of complex changes needed to improve pages in real life.
Can it detect "spin"? (Score:2)
Detecting spam-like vandalism would seem to be fairly easy.
Far more insidious is politically spun issue-framing masquerading as objective description
of events or topics.
It is truly amazing what you can hide in there by using high-falutin', officious,
grammatically correct language to accomplish your spin. Oft' times you can even fool the
domain experts.
Physicists say that everything is either "spin-UP" or "spin-DOWN". Master spin-doctors
say the same thing.
Simple (Score:1)
Why? (Score:2)
Why vandalize articles in the first place?
Sure, stupid spammers think replacing an article with a badly spelled advert for ViAGRa is the way to go, and morons think that they gain something from inserting "I'M GAY!!!!!" into an article about someone they dislike, but why just do damage for no other purpose than destroying other people's hard work?
I just don't get it.
These trolls/vandals need to get their asses kicked - hard. Or maybe just have something of theirs broken, just for the fun of it, and see if t