Open-Source Python Code Shows Lowest Defect Density 187
cold fjord sends news that a study by Coverity has found open-source Python code to contain a lower defect density than any other language. "The 2012 Scan Report found an average defect density of .69 for open source software projects that leverage the Coverity Scan service, as compared to the accepted industry standard defect density for good quality software of 1.0. Python's defect density of .005 significantly surpasses this standard, and introduces a new level of quality for open source software. To date, the Coverity Scan service has analyzed nearly 400,000 lines of Python code and identified 996 new defects — 860 of which have been fixed by the Python community."
Coverity fails to detect errors in python (Score:4, Insightful)
"Coverity fails to detect errors in python" would be my headline of choice here. Seem a much more reasonable explanation for the results.
Re: (Score:2)
"Coverity fails to detect errors in python" would be my headline of choice here. Seem a much more reasonable explanation for the results.
Or, to put it another way, "static analysis tool fails to detect many potential errors in code whose authors use the same static analysis tool to find and fix potential errors." Which is hardly surprising.
Re: (Score:2)
The actual doc says "Some aspects of Python’s C code are not yet understood by Coverity." That's much more vague admittedly, but not as shameful.
Re: (Score:3)
This causes false positives, so if they are really not bugs, then Python's code is even more awesome :D
Re: (Score:2)
Some would argue that having a codebase that's so hard to understand that static analysis tools get confused about what it does is a bug in itself.
Can some one please explain? (Score:3)
I read TFS and both TFAs and all I can glean is that Coverity Scan service is some sort of report that measures defects in code, but never defines how such defect are determined. They articles also mention comparing open source code metrics, but the only project that is mentioned anywhere is Python.
So what is a Coverity Scan service and why should I care? After all I can make up all sorts of metrics about my own software.
Re: (Score:2)
What is Coverity Scan service? It is a product they hope to sell you. Does advertising work? It just did!
Re: (Score:3)
A defect is when the code uses encryption, and doesn't send the keys to the NSA, or uses smtplib, and doesn't bcc:archives@dea.gov.
Re:Can some one please explain? (Score:4, Informative)
Re: (Score:2)
So what is a Coverity Scan service
It's the same idea as the 'lint' command, it picks up potential bugs.
These sort of tools can't help improve the quality of your code. Having said that, in my (20+) years of experience it's not common practice to use these things, I've worked on several large "mission critical" systems and the Y2K ordeal was the only time someone even asked if I used such a tool, let alone demanded it. At the end of the day (actually more like a month) the "Y2K lint" tool's only practical achievement was to tick a due-dil
Re: (Score:2)
if it's the same as lint then eh..
of course it has less "defects" to complain about.. with all that whitespace shit defined in language and all.
"style errors" aren't defects. they're just matter of deciding who decides the right style..
Re: (Score:2)
Warning: pdf
http://wpcme.coverity.com/wp-content/uploads/2012-Coverity-Scan-Report.pdf [coverity.com]
explains much if not all that you ask
For a good article and a fun read that goes into the background of Coverity and what it does, see
http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext [acm.org]
it's written by some of the developers and founders
Where is the study? (Score:2, Informative)
I could not find a link to the actual study, instead the company links lead back to the article and the article leads back to the company home page. Is this more "faith-based computing"? I am interested in the comparisons to other languages and in what type of code was analyzed.
Hmmm (Score:5, Informative)
TFA seems to be about the Python interpreter, also known as CPython (because it's implemented in C), rather than about code written in Python itself. So maybe it has nothing to do with the Python language, but everything to do with the fact that the Python authors are apparently awesome C programmers.
That's great, but most people interpret "Open Source Python Code" to mean code written in Python that is Open Source, not code written in C (to implement the Python interpreter) that is Open Source.
Re: (Score:2)
Re: (Score:2)
The nice thing about Python interpreter is that it is deliberately written in such a way as to be easy to read and understand. I suspect they could have squeezed quite a bit more performance out of it by using more exotic techniques (e.g. tagged ints), but, arguably, it is not worth it - if you want real perf, you'd do JIT anyway (and that's what PyPy is for), while on the other hand it is beneficial to have a well-understood, stable and foolproof reference implementation for the language.
ok, and this means what? (Score:2)
Does it mean better coders, or better language? Seems like the results are ambiguous in their meaning.
Re: (Score:2)
This is w/r/t CPython, not random code in Python (Score:5, Informative)
The Slashdot summary is confusing, as is the eweek.com headline. Reading the article, it is clear that it is about the code that powers the official Python interpreter, AKA CPython, AKA /usr/bin/python. When I clicked the link, I thought Coverity had surveyed the entire world of open source Python code and discovered that Python programmers as a whole publish higher quality code than people who e.g. program in Ruby. That's not what the article's about.
It'd be great if the headline in Slashdot were to be fixed to say, "Python interpreter has fewer code defects compared to other open source C programs, says Coverity."
Re: (Score:2)
That makes more sense. From the summary, I thought the most likely scenario was that Coverity does not handle Python code very well based on my experience of random buggy Python code. It is to be expected that a widely used VM/interpreter is going to be of better quality than your average code.
Re: (Score:2)
Math impairment (Score:5, Informative)
0.005 defects per thousand lines times 400,000 lines gives a total defect count of 2.
So where did the other 994 defects come from?
Re: (Score:2)
0.005 defects per thousand lines times 400,000 lines gives a total defect count of 2.
So where did the other 994 defects come from?
They were in comments.
Re: (Score:2)
I'm more interested in this software that detects bugs in code. Does it also solve the halting problem? Can it satisfy finite combinational logic in polynomial time?
Re: (Score:3)
I'm more interested in this software that detects bugs in code. Does it also solve the halting problem? Can it satisfy finite combinational logic in polynomial time?
The don't claim to find all bugs. I have used Coverity, and they found quite a few bugs, and also found many instances of unclear code that wasn't really a bug but should be rewritten anyway. But they don't find most logic bugs, or flaws in your requirements, etc. You still have to use your brain for those. But you can use tools like Coverity and other dynamic and static analysis tools to flag the easy bugs so you can spend more time on the hard bugs.
Re: (Score:2)
Does it analyze source code or is it like a fuzz tester?
Re:Math impairment (Score:5, Informative)
Does it analyze source code or is it like a fuzz tester?
It is static analysis of source code. It doesn't actually run the code, it scans it for patterns that might be bugs. I like Gimpel Lint [wikipedia.org] better, but it isn't either-or, so you can use both and they will find different bugs. You still need to do dynamic testing with something like Valgrind [wikipedia.org]. Tools are cheap compared to people, so you want to give your developers the best testing tools you can, and put your code through the wringer. We use six different tools for C/C++, and no code is shipped out the door till it passes them all (plus unit, usability, and requirements testing).
Re: (Score:2)
I wish I could mod you informative for your response.
Re: (Score:3)
and no code is shipped out the door till it passes them all
I quite agree. I won't ship my code until it passes the test tool I use. My test tool is gcc. Once that runs without error, I ship.
Re: (Score:3)
However, only 860 were fixed. Double logic impairment.
Re:Math impairment (Score:5, Funny)
Maybe those two LOC are is really, really, really bad.
Re: (Score:2)
So their index is not defects per 1000 LOC, as the GP assumed, but defects per 2.03 lines of code. I guess they had to change the factor after a few high level managers at large corporates ran their department's code through it after committing to certain KPI targets.
Excellent marketing! (Score:5, Insightful)
Re: (Score:2)
So a private, for-profit company named "Coverity" has released a report that shows that their "Coverity Scan" software finds the fewest vaguely-defined "defects" in a programming language whose community has added the "Coverity platform" product to their development process?
Their stuff does work at detecting certain kinds of problem, but it doesn't detect all possible bugs (nor does anything else I've encountered). It's better to say that it's an independent tool that can be used as well as other tools, and they provide free access to quite a few of the larger OSS projects. They surely don't have to; nobody's forcing them. They've also been doing it for years.
For an example of the sort of thing they find, in a software package I know about their tool recently picked up that th
Re: (Score:2)
Coverity: Static analyzer (Score:5, Informative)
Coverity sells software that does static analysis on source code and looks for patterns that suggest defects. E.G., a code sequence that allocates memory, followed later by something that de-allocates that memory, followed later by something that de-allocates the same memory again (a double-free).
The product is not open source software, but a number of open source software projects use it to scan their software to find defects: https://scan.coverity.com/ [coverity.com] It's a win-win, in the sense that Coverity gets reports from real users using it on real code, as well as press for their product. The open source software projects get reports on potential defects before users have to suffer with them.
Re: (Score:3, Interesting)
We've ran Coverity on several very large projects where I work. For C++ it did a decent job of finding little and simple things that Visual Studio missed, like variables that were never initialized before use, subtle type violations Visual Studio missed, or accessing past the end of a statically allocated array. These aren't the sorts of bugs that we worry about. The evil bugs - like those created by programmers that don't know enough about multithreading but were assigned because some offshore contracto
Re: (Score:2, Informative)
you should try TSAN. See : https://code.google.com/p/thread-sanitizer/
Past Coverity reviews (Score:5, Informative)
Coverity's services have been useful to a number of open-source projects. But this article is carefully picking its terms to get a headline worthy result. Compare against the Coverity scan of PostgreSQL [postgresql.org] done in 2005 for example, and CPython's defect rate isn't very exciting at all. But that was "Coverity Prevent" and this is "Coverity Scan"...whatever that means.
Re: (Score:2)
I tried to be clear that the two results can't be directly compared. My main point is that Coverity likes to put open-source projects in a good light, because there's better PR value for them to do so. Any sort of "best project evuh!" claims from them should recognize that this is ad copy designed to draw attention with its superlatives.
C code, not Python code (Score:2)
How rude! (Score:3)
They counted my C++ features as bugs?
What does the measering mean? (Score:2)
Numbers like .69 or 1.0 or 0.005 mean nothing if you don't know to what it relates.
Usually defect counts are based on 1k LOC (one thousand lines of code, and no: a line of code is likely not what you consider a line of code).
I doubt that 1.0 is a accepted industry standard defect density [...] for good quality software of ...
1 defect per 1 kLOC is absurd high, luckily I never was in a project the last 20 years with such a high defect rate.
Re: (Score:2)
Re: (Score:2)
Yeah, what exactly is considered a defect varies.
In the personal software process by Watts Humphrey(sp) already a line that does not compile is considered a defect and is added to the defect log.
Bottom line everything that comes up in an issue tracker with the aim to fix it later, is a defect.
In that regard, sleeping defects that are never discovered because "never" some invalid data triggers them, are no defects.
Regarding 1 error per kLOC. Serious tools count something like this:
Re: (Score:2)
Regarding your Input example: I disagree. Most enterprise systems are very good in rejecting invalid input.
I think you probably misunderstood what I meant by "invalid inputs". Take the infamous example of the 32-bit version of the Java's binary search in a sorted array [blogspot.cz]: the problem was in the overflow of the midpoint computation: while (a+b)/2 looks like a reasonable way to do it, even if both a, b, and (a+b)/2 are within the range of the integer type used, a+b doesn't necessarily have to in some cases. But since few people did multi-GB arrays to even potentially get the <a,b> tuple into an invalid range,
Hey metric retards (Score:4, Interesting)
Re: (Score:2)
According to their report (take it as you will) false positives as of 2012 were 9.7% of reported defects.
Re: (Score:2)
Would you care to share your justification for submitting a story with a grossly misleading headline and story? The code analysis in question wasn't performed on software written in the Python programming language; it was performed on the Python interpreter written in C. Again, why would you submit a story under such horrendously misleading premises? You've probably caused a pile of headaches for developers who will have to explain the difference between C and Python to their development "managers."
Re: (Score:2)
You also managed to miss the GP's point that in his experience, most of the "defects" aren't actually defects at all, but false positives that result from Coverity depending on a certain coding style.
Bullshit (Score:2)
This is bullshit, but a great tactical conversion of non-informative data into marketable news by Coverity.
Coverity uses lexical pattern matching to find bugs based on "tricks" discovered by Dawson Engler and his colleagues in Stanford University in the early 2000s. The tricks (find "malloc" not coupled with "free", cli() not coupled with sti(), dereferences of uninitialized pointers etc.) were developed in the context of the C language used for Operating System code.
So they used tricks developed for one la
Re: (Score:3)
Re: (Score:2)
..and thats why there are few job opportunities. (Score:2)
Re: (Score:2)
On the other hand, there are also proportionally many Java and .NET programmers, so you'll be competing with fewer people in Python land.
The right answer, anyway, is to learn all three - and a couple more (C++, in particular).
They've lots of time.. (Score:2)
The code is so slow, they have lots of extra time to look for defects.
Re: (Score:2)
When you look at analyzing defects - you can find coding defects pretty easily but you can't find design defects where the designer has misunderstood the goal of the product.
One example of a pretty annoying design mistake is when you run Microsoft software where you can chose to send a document as an attachment from Powerpoint, Excel or Word. However it will at the same time block all access to other windows in Outlook preventing you to get the list of names that you know were present in another message. No
uh, i RTFA and... (Score:2)
What this means (Score:2)
So what they are basically saying is "Don't use our product to scan Python code; it doesn't recognize all the defects".
I know the truth is possibly somewhere in the middle, but this report just assumes the scanning products works equally well for all languages, which is atleast somewhat unlikely.
Also, what exactly is a defect in this context? Is it a security flaw, a functional error or just something that will crash your software. If the latter is the case, then any language that accepts shitty code and ju
Misleading (Score:2)
Re:Python is readable (Score:5, Funny)
Python is readable and readable code is easier to fix.
Also smarter guy have tendency to use Python/Haskell/Erlang
Oh yeah? Well, I'm working on a readable Perl script to refute that statement. How long do they accept comments in these threads?
Re: (Score:2)
I've seen multiple-kilobyte posts before. Slashdot truncates it on initial display with a 'read more' link appended to the end, that shows the full post.
Re: (Score:2)
Re:Python is readable (Score:4, Funny)
Python is readable and readable code is easier to fix.
Also smarter guy have tendency to use Python/Haskell/Erlang
Oh yeah? Well, I'm working on a readable Perl script to refute that statement. How long do they accept comments in these threads?
How is this possible? Perl is a write only language.
Perl IS readable (Score:3, Funny)
@*(&^)&^)^$
Perl programmers write their code in cartoon profanity!
Re: (Score:2)
If you write a readable Perl script, then you've completely missed the point of the language. Ever hear of job security?
Re: (Score:2)
Re: (Score:2)
... which worked fine when I ran it.
Re: (Score:2)
Re:Python is readable (Score:5, Informative)
The result in question tested the Python project's code, which is commonly known as CPython, which is the Python interpreter written in C.
Re: (Score:3)
It appears you're right. Neither the submitter nor the article writer understand the difference between "code written in Python" and "the CPython interpreter, which is written in C", which is what Coverity actually tested. So 90% of the comments are off topic. Mods - kudos to the parent.
Re:Python is readable (Score:4, Insightful)
Python is readable and readable code is easier to fix.
True and true. But Python's use of semantic whitespace is also very brittle very easy to break, and a huge pain in the ass to fix compared to languages that use braces, or keywords to define 'blocks'.
But that's not even terribly relevant here, because this article is about the source code used for the python interpreter, which is C, not python.
Re:Python is readable (Score:5, Insightful)
But Python's use of semantic whitespace is also very brittle very easy to break, and a huge pain in the ass to fix compared to languages that use braces, or keywords to define 'blocks'.
This is one thing I never quite get about python criticism. Sure, whitespace is significant, but I've never had it break easily or be "brittle" as you say. Then again, I don't go past 2 or 3 levels of nesting, class nesting included. And all my units of work are in separate methods/functions instead of being child blocks inside a giant function which I've regularly seen done. Perhaps the use of whitespace isn't the real issue many people have with python, but rather delineating blocks using whitespace exposes a bit of an inherent flaw in the way they structure their program's flow.
Either way, having a proper IDE when writing python code will go a long way to making you comfortable with using whitespace instead of braces. Initially it was weird and unsettling for me, because I didn't understand all the consequences that whitespace could have. But a little fluid and constant coding in a IDE will rid you of that quick enough.
Re: (Score:2)
I saw a trivial example break when posted to /. not that long ago, in the interview.
Re: (Score:2)
Although, I have been bitten many times when copy-pasting python code between a text file and the command line. Though I've mostly gotten around that problem by working with files rather than trying to use the CLI to input arbitrary python code as every single console does it slightly differently.
Re:Python is readable (Score:5, Insightful)
Sure, whitespace is significant, but I've never had it break easily or be "brittle" as you say.
Not python, but one example of this type of thing would be in a Makefile where target commands are indented by a tab. Some newer versions of (g)make will allow spaces, but most require a tab. Cut and paste that in an X-Windows session (tabs are converted to spaces) and you're screwed. From Make Software: Makefiles [wikipedia.org]
Each command line must begin with a tab character to be recognized as a command. The tab is a whitespace character, but the space character does not have the same special meaning. This is problematic, since there may be no visual difference between a tab and a series of space characters. This aspect of the syntax of makefiles is often subject to criticism.
Re:Python is readable (Score:5, Insightful)
This is one thing I never quite get about python criticism. Sure, whitespace is significant, but I've never had it break easily or be "brittle" as you say.
Anytime you refactor stuff, or modify something even somewhat nested, especially in a 'dumb text editor', it's a pain in the ass.
Anytime you need to pass code snippets via email, forums, etc... well... you just don't because its a total waste of time. :)
Its also easy to barf all over code going into word processors, pdf files, and so forth. Its nice to be able to copy-paste some C out of a PDF file or an email, or off a forum, and then tell the ide to just reformat it.
erhaps the use of whitespace isn't the real issue many people have with python, but rather delineating blocks using whitespace exposes a bit of an inherent flaw in the way they structure their program's flow.
No. Because we use whitespace / indenting in our C / C++ etc projects too. We even have standards requiring it, and our IDEs / toolchains may even be set up to reformat it just-so before commits. We want all the benefits of well formatted code.
We just like the IDE to do all the work actually formatting it, and reformatting it as neccessary.
Either way, having a proper IDE
Is how you lose the argument. Everyone but python groupies agrees that any programming language worth considering MUST have its programs represented as plaintext files, with no proprietary / binary stuff that can only be accessed with specialized tools. Requiring an IDE is the sign of a bad language.
Python passes this test, but it can be pretty hideous to use with an arbitrary text editor. And really, even brainfuck wouldn't be too bad with the right IDE, right?
Re: (Score:3)
Is how you lose the argument. Everyone but python groupies agrees that any programming language worth considering MUST have its programs represented as plaintext files, with no proprietary / binary stuff that can only be accessed with specialized tools. Requiring an IDE is the sign of a bad language.
I don't think you understood what I was trying to say here. The IDE is there to teach you the boundaries when it comes to whitespace in python. Bad indentation, mismatching brackets and overall bad syntax gets picked up immediately and you are warned. Just like you get syntax error highlighting in other languages. Python's usage of whitespace scares a lot of people and keeps them from experimenting. The IDE is what I think would help them overcome their fear/uncertainty. If anything, Python is one of the la
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
*cough*
The code reformatting can be done manually, via a command line tool, via the IDE or not at all. My mention of using an IDE to reformat text doesn't create the same IDE dependency you refer to.
Re:Python is readable (Score:4, Insightful)
Blaming Python because you don't have a rudimentary coding editor is like blaming math because you don't have a calculator with a cosine button.
I don't have the luxury of designing "math". Irrational numbers, periodic functions, and so forth aren't optional.
But we do have the luxury of designing programming languages, and semantic white space is a choice.
Re: (Score:2)
If you're using a "dumb" text editor, then don't complain about it.
Your argument is back-to-front. Python has whitespace because of dumb editors. Guido's rationale was simple: when writing C in a dumb editor, there is redundancy of braces (for the computer) and spaces (for the human). There is the danger that the two might not match, and that a human debugging the code would misread the structure by following the indentation levels instead of the braces.
And here lies the problem: Guido's decision was for the sake of "plain text" and dumb editors, but the end result was to
Re: (Score:2)
Sure, whitespace is significant, but I've never had it break easily or be "brittle" as you say
The Jabber Python MSN transport shipped with an intent bug in an error path for several releases. The error path was never hit on the developer's test machine, but always hit for me because I didn't install one of the optional libraries. The error was caused by mixing tabs and spaces, and so looked correct in the editor, but Python happened to interpret a tab as a different number of spaces to the editor[1] and so it ended up doing something different.
This is what people mean when they call it fragile.
Re: (Score:2)
I've been asking Python programmers for the last year what an else clause on a for loop meant. Last Friday, one gave the correct answer for the first time. Why do I know what it means? Because a person who wrote some (and shipped) some code using it apparently didn't...
I didn't know that structure... it should be banned... it's totally "un-pythonic" in that it annihilates the principle of readability. Kill it with fire.
Re: (Score:2)
Re: (Score:2)
Python is readable and readable code is easier to fix.
True and true. But Python's use of semantic whitespace is also very brittle very easy to break, and a huge pain in the ass to fix compared to languages that use braces, or keywords to define 'blocks'.
Furthermore Python's needless attribution of syntactical meaning to whitespace means it's useless for embedding certain languages...
...Like Whitespace. [dur.ac.uk]
Today many languages support Unicode source code which can have tons of new spaces of varying width including zero-width and non-breaking-zero-width space. The multitude of new spaces would make indention distinction all the more brittle, but this also means new extensions to Whitespace can provide more rich and full featured embedded language support to
Re: (Score:2)
Re: Python == LAME (Score:5, Informative)
Most of Python isn't written in Python, smart ass. They're talking about the language interpreter itself, written in C/C++ etc.
Re:WRONG! RTFA! (Score:5, Insightful)
Great. Now where the hell do you quote it from, since that sure as hell isn't in the linked to article anywhere.
That makes it pretty clear that they are talking about the Python executable itself. Version 3.3.2 to be exact.
... and that clearly shows that they are talking about the interpreter, written in C, which has pointers, malloc() and free(). Python has a memory manager with garbage collection and doesn't use pointers. The Python programmer doesn't allocate and free memory resources directly.
I especially love how you criticized a language earlier, when you clearly have literally no knowledge of said language.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
What about the bit about Coverity Scan only supporting Java and C/C++?
Re:Python == LAME (Score:4, Interesting)
Nope, nobody at all http://www.python.org/about/success/ [python.org]
Jeez.
Python one-liner (Score:2)
Re: (Score:2)
it might have an advantage in forcing lazy programmers with no concept of 'code etiquette' to write semi-readable code as indentation is forced by syntax.
Since the "density" is measured in defects per lines of code, I siggest that Python mandate an extra line return between all lines. Then they could half their defect density. Done.
Re: (Score:2)
int
my_int
C initializers (Score:2)
Re: (Score:2)
Re:Can't be right (Score:4, Informative)
it might have an advantage in forcing lazy programmers with no concept of 'code etiquette' to write semi-readable code as indentation is forced by syntax.
on the other hand, making indentation part of the language creates all sorts of other readability problems.
You'd be surprised at how much syntax in python actively ignores whitespace. As soon as you open up any brackets, it's a veritable free-for-all when it comes to whitespace and indentation. In such a scenario, a proper coding standard document is imperative for readable code.