Forgot your password?
typodupeerror
Programming IT Technology

Do Static Source Code Analysis Tools Really Work? 345

Posted by CmdrTaco
from the if-you're-stupid-they-do dept.
jlunavtgrad writes "I recently attended an embedded engineering conference and was surprised at how many vendors were selling tools to analyze source code and scan for bugs, without ever running the code. These static software analysis tools claim they can catch NULL pointer dereferences, buffer overflow vulnerabilities, race conditions and memory leaks. Ive heard of Lint and its limitations, but it seems that this newer generation of tools could change the face of software development. Or, could this be just another trend? Has anyone in the Slashdot community used similar tools on their code? What kind of changes did the tools bring about in your testing cycle? And most importantly, did the results justify the expense?"
This discussion has been archived. No new comments can be posted.

Do Static Source Code Analysis Tools Really Work?

Comments Filter:
  • Re:In Short, Yes (Score:3, Interesting)

    by tritonman (998572) on Monday May 19, 2008 @12:28PM (#23463794)
    yea they work, I wouldn't spend a lot of money on them though, a decent compiler will give you info for stuff like buffer overflows. Most of the bugs will need to be caught at runtime, so if you are on a tight budget, definitely skip the static tools for the more useful ones.
  • by MadShark (50912) on Monday May 19, 2008 @12:29PM (#23463802)
    I use PC-lint religiously for my embedded code. In my opinion it has the most bang for the buck. It is fast, cheap and reliable. I've found probably thousands of issues and potential issues over the years using it.

    I've also used Polyspace. In my opinion, it is expensive, slow, can't handle some constructs well and has a *horrible* signal to noise ratio. There is also no mechanism for silencing warnings in future runs of the tool(like the -e flag in lint). On the other hand, it has caught a (very) few issues that PC-Lint missed. Is it worth it? I suppose it depends if you are writing systems that can kill people if something goes wrong.
  • They do work (Score:5, Interesting)

    by Anonymous Coward on Monday May 19, 2008 @12:29PM (#23463804)
    Static analysis does catch a lot of bugs. Mind you, it's no silver bullet, and frankly it's better, given the choice, to target a language+environment that doesn't suffer problems like dangling pointers in the first place (null pointers, however, don't seem to be anything Java or C# are really interested in getting rid of).

    Even lint is decent -- the trick is just using it in the first place. As for expense, if you have more than, oh, 3 developers, they pay for themselves by your first release. Besides, many good tools such as valgrind are free (valgrind isn't static, but it's still useful).
  • by wfstanle (1188751) on Monday May 19, 2008 @12:32PM (#23463824)
    I am presently working on an update to static analysis tools. Static analysis tools are not a silver bullet but they are still relevant. Look at them as a starting point in your search for programming problems. A lot of potential anomalies can be detected like the use of uninitialized variables. Of course, a good compiler can use these tools as part of the compilation process. However, there are many things that a static analyzer can't detect. For this, you need some way to do dynamic analysis ( execution based testing). As such the tools we are developing also include dynamic testing.
  • by Idaho (12907) on Monday May 19, 2008 @12:33PM (#23463840)
    Such tools work in a very similar way to what is already being done in many modern language compilers (such as javac). Basically, they implement semantic checks that verify whether the program makes sense, or is likely to work as intended in some respect. For example, they will check for likely security flaws, memory management/leaking or synchronisation issues (deadlock, access to shared data outside critical sections, etc.), or other kind of checks that depend on whatever domain the tool is intended for.

    It would probably be more useful if you could state which kind of problem you are trying to solve and which tools you are considering to buy. That way, people who have experience with them could suggest which work best :)
  • MIT Site (Score:4, Interesting)

    by yumyum (168683) on Monday May 19, 2008 @12:38PM (#23463890)
    I took a very cool graduate-level class at MIT from Dr. Michael Ernst [mit.edu] about this very subject. Check out some of the projects listed at http://groups.csail.mit.edu/pag/ [mit.edu].
  • Very useful in .Net (Score:3, Interesting)

    by Toreo asesino (951231) on Monday May 19, 2008 @12:39PM (#23463904) Journal
    While not the be-all-and-end-all of code quality metrics, VS2008/Team Foundation Server has this built-in now so you can stop developers checking in completely junk code if you so wish - http://blogs.msdn.com/fxcop/archive/2007/10/03/new-for-visual-studio-2008-code-metrics.aspx [msdn.com]

    FxCop too has gone server-side too (for those familiar with .Net development). It takes one experienced dev to customise the rules, and you've got a fairly decent protection scheme against insane code commits.
  • by RJCantrell (1290054) on Monday May 19, 2008 @12:40PM (#23463918)
    In my own corner of the world (.NET Compact Framework 2.0 on old, arcane hardware), they certainly don't. Each time I get optimistic and search for new or previously-missed static analysis tools, all roads end up leading to FxCop. Horrible signal-to-noise ratio, and a relatively small number of real detectable problems. That said, I'm always willing to submit myself to the genius of the slashdot masses. If you know of a great one, feel free to let me know. = )
  • by Chairboy (88841) on Monday May 19, 2008 @12:41PM (#23463938) Homepage
    At Symantec, I used to use these tools to help plan tests. I wrote a simple code velocity tool that monitored Perforce checkins and generated code velocity graphs and alerts in different components as time passed. With it, QA could easily see which code was being touched the most and dig down to the specific changelists and see what was going on. It really helped keep good visibility on what needed the most attention and helped everyone avoid being 'surprised' by someone dropping a bunch of changes into an area that wasn't watched carefully. During the final days of development before our products escaped to manufacturing, this provided vital insight into what was happening.

    I've since moved on, and I think the tool has since gone offline, but I think there's a real value to doing static analysis as part of the planning for everything else.
  • Trends or Crutches? (Score:4, Interesting)

    by bsDaemon (87307) on Monday May 19, 2008 @12:51PM (#23464066)
    I'll probably get modded to hell for asking but seriously -- all these new trends, tools, etc - are they not just crutches, which in the long run are seriously going to diminish the quality of output by programmers?

    For instance, we put men on the moon with a pencil and a slide rule. Now no one would dream of taking a high school math class with anything less than a TI-83+.

    Languages like Java and C# are being hailed while languages like C are derided and many posts here on slashdot call it outmoded and say it should be done away with, yet Java and C# are built using C.

    It seems to me that there is no substitute for actually knowing how things work at the most basic level and doing them by hand. Can a tool like Lint help? Yes. Will it catch everything? Likely not.

    As generations of kids grow up with the automation made by generations who came before, and have less incentive to learn how the basic tools work, an incentive which will diminish, approaching 0, I think we're in for something bad.

    As much as people bitch about kids who were spoiled by BASIC, you'd think that they'd also complain about all the other spoilers. Someday all this new, fancy stuff could break and someone who only knows Java, and even then checks all their source with automated tools will likely not be able to fix it.

    Of course, this is more of just a general criticism and something I've been thinking about for a few weeks now. Anyway, carry on.
  • To a degree, yes (Score:5, Interesting)

    by gweihir (88907) on Monday May 19, 2008 @12:53PM (#23464098)
    You actually need to tolerate a number of false positives in order to get good coverage of the true bugs. That means you have to follow-up on every report in detail and understand it.

    However these things do work and are highly recommended. If you use other advanced techniques (like Descign by Contract),they will be a lot less useful though. They are best for traditional code that does not have safety-nets (i.e. most code).

    Stay away from tools that do this without using your compiler. I recently evaluated some static analysis tools found that the tools that do not use the native compilers can have serious problems. One example was an incorrecly set symbol in the internal compiler of one tool, that could easily change the code functionality drastically. Use tools that work frrom a build environment and utilize the compiler you are using to build.
  • Re:Yes (Score:3, Interesting)

    by Pollardito (781263) on Monday May 19, 2008 @12:55PM (#23464116)

    These tools take static checking just a step beyond what's offered by a compiler, but in practice that's very useful.
    that's a good point in that compiler warnings and errors are really just the result of static analysis, and i think everyone has experience in finding bugs due to those
  • Buyer (User) Beware (Score:3, Interesting)

    by normanjd (1290602) on Monday May 19, 2008 @01:05PM (#23464228) Homepage
    We use them more for optimizing code than anything else... The biggest problem we see is that there are often false positives... A senior person can easily look at recomendations and pick whats needed... A junior person, not so much, which we learned the hard way...
  • Re:In Short, Yes (Score:5, Interesting)

    by FBSoftware (1224962) on Monday May 19, 2008 @01:13PM (#23464308)
    Yes, I use the formal methods based SPARK tools (www.sparkada.com) for Ada software. In my experience, the Examiner (static analyzer) is always right (> 99.44% of the time) when it reports a problem or potential for runtime exception. Even without SPARK, the Ada language requires that the compiler itself accomplish quite a bit of static analysis. Using Ada, its less likely you will need third-party static analysis tool - just use a good compiler like GNAT.
  • by bsDaemon (87307) on Monday May 19, 2008 @01:15PM (#23464336)
    In the 7th grade I left my calculator at home one day when I had a math test. I did, however, have a Jepsen Flight Computer (a circular slide rule) that my dad (a commercial airline pilot) had given me, because I was going to a flying lesson after school.

    I whipped out my trusty slide rule and commenced to using it. The teacher wanted to confiscate it and thought that I was cheating with some sort of high-tech device... mind you it was just plastic and cardboard. I'm sure you've all seen one before.

    I'm only just about to turn 24, so 7th grade was not long ago for me.

    The point is, students should be required to know how to do things by hand. a PhD in physics 20 years ago clearly knew how to do calculus by hand. if he wants to use a Ti-92 to do it now, that's his business. Good that those aren't allowed in class (even if they are allowed on the SAT and AP exams, well the ti-89 is).

    Intro to comp sci shouldn't be taught with Java any more than elementary school math should be "intro to the calculator." You just cripple people's minds that way.

    I ended up getting a BA in English the first time around because of personal reasons I was too messed up to concentrate on maths. I'm now starting to take a 2nd degree in MechE - and I'm making a good faith effort to do as much by hand as possible, because I don't want to fuck something up in the future because I figured the calculator was giving me a right answer and I had no idea of where the proper answer should be.

    summary: using a tool to check your answer is one thing. Relying on it to get the answer in the first place is lazy, stupid, and potentially dangerous.
  • Absolutely (Score:2, Interesting)

    by Fippy Darkpaw (1269608) on Monday May 19, 2008 @01:18PM (#23464372)
    At my little corner of Lockheed Martin we use Klocwork [klocwork.com] and LDRA [ldra.com] to analyze C/C++ embedded code for military hardware. Since the various compilers for each contract aren't nearly as full-featured as say, Visual Studio or Eclipse, I've found static code analysis tools invaluable. Can't comment on the cost/results ratio though, since I don't purchase stuff. =)
  • WHOA... nice timing (Score:2, Interesting)

    by w00f (872376) on Monday May 19, 2008 @01:23PM (#23464416)
    YOU sir, have amazing timing! I just wrote a 2-part article on this topic! Interesting... mine was published http://portal.spidynamics.com/blogs/rafal/archive/2008/05/06/Static-Code-Analysis-Failures.aspx The Solution: http://portal.spidynamics.com/blogs/rafal/archive/2008/05/15/Hybrid-Analysis-_2D00_-The-Answer-to-Static-Code-Analysis-Shortcomings.aspx [spidynamics.com] Comments welcome!! Interesting that this topic is getting so much attention all of the sudden
  • by nguy (1207026) on Monday May 19, 2008 @01:25PM (#23464448)
    Generally, these tools make up for deficiencies in the underlying languages; better languages can guarantee absence of these errors through their type systems and other constructs. Furthermore, these tools can't give you yes/no answers, they only warn you about potential sources of problems, and many of those warnings are spurious.

    I've never gotten anything useful out of these tools. Generally, encapsulating unsafe operations, assertions, unit testing, and using valgrind, seem both necessary and sufficient for reliably eliminating bugs in C++. And whenever I can, I simply use better languages.
  • by tikal_work (885055) on Monday May 19, 2008 @01:36PM (#23464566)

    Something that we've found incredibly useful here and in past workplaces was to watch the _differences_ between Gimpel PC-Lint runs, rather than just the whole output.

    The output for one of our projects, even with custom error suppression and a large number of "fixups" for lint, borders on 120MiB of text. But you can quickly reduce this to a "status report" consisting of statistics about the number of errors -- and with a line-number-aware diff tool, report just any new stuff of interest. It's easy to flag common categories of problems for your engine to raise these to the top of the notification e-mails.

    Keeping all this data around (it's text, it compresses really well) allows you to mine it in the future. We've had several cases where Lint caught wind of something early on, but it was lost in the noise or a rush to get a milestone out -- when we find and fix it, we're able to quickly audit old lint reports both for when it was introduced and also if there are indicators that it's happening in other places.

    And you can do some fun things like do analysis of types of warnings generated by author, etc -- play games with yourself to lower your lint "score" over time...

    The big thing is keeping a bit of time for maintenance (not more than an hour a week, at this point) so that the signal/noise ratio of the diffs and stats reports that are mailed out stays high. Talking to your developers about what they like / don't like and tailoring the reports over time helps a lot -- and it's an opportunity to get some surreptitious programming language education done, too.

  • Re:In Short, Yes (Score:3, Interesting)

    by samkass (174571) on Monday May 19, 2008 @01:56PM (#23464790) Homepage Journal
    Due to its dynamic nature and intermediate bytecode, Java analysis tools seem to be especially adept at catching problems. In essence, they can not only analyze the source better (because of Java's simpler syntax), they can also much more easily analyze swaths of the object code and tie it to specific source file issues.

    In particular, I've found FindBugs has an amazing degree of precision considering it's an automated tool. If it comes up with a "red" error, it's almost certainly something that should be changed. I'm not familiar with any C/C++ tool that comes close.
  • by EPAstor (933084) on Monday May 19, 2008 @02:13PM (#23465026)

    I did some work running Coverity for EnterpriseDB, against the PostgreSQL code base (and yes, we submitted all patches back, all of which were committed).

    Based on my experience:

    1) Yes, Coverity produced a LOT of false positives - a few tens of thousands for the 20-odd true critical bugs we found. However, the first step in working with Coverity is configuring it to know what can be safely ignored. After about 2 days of customizing the configuration (including points where I could configure it to understand that certain methods fixed its common complaints already), the bug list dropped to 200. 2 days of configuration by an inexperienced user dropped it to a total false positive rate of about 75%, if that, though only about half of those proved to be critical bugs. I think that's good enough to be useful.

    2) Can't argue - horrendously expensive.

    3) Postgres is smaller than what you're talking about... but still, the speed was a pain, but the system was useful enough to make up for it, particularly once we had the system configured to know what bugs were real and automated its running every few nights. While EnterpriseDB still had the license, it was a great supplement to nightly testing.

  • Re:In short, YMMV (Score:4, Interesting)

    by JaneTheIgnorantSlut (1265300) on Monday May 19, 2008 @02:26PM (#23465156)
    Also beware of managers who insist that each item identified by the tool needs to be somehow addressed. I inherited a body of code full of comments to the effect that "the tool says this is a a problem, but I looked at it and it is not".
  • by Animats (122034) on Monday May 19, 2008 @03:22PM (#23465834) Homepage

    Several posters have cited the "halting problem" as an issue. It's not.

    First, the halting problem does not apply to deterministic systems with finite memory. In a deterministic system with finite memory, eventually you must repeat a state, or halt. So that disposes of the theoretical objection.

    In practice, deciding halting isn't that hard. The general idea is that you have to find some "measure" of each loop which is an integer, gets smaller with each loop iteration, and never goes negative. If you can come up with a measure expression for which all those properties are true, you have proved termination. If you can't, the program is probably broken anyway. Yes, it's possible to write loops for which proof of termination is very hard. Few such programs are useful. I've actually encountered only one in a long career, the termination condition for the GJK algorithm for collision detection of convex polyhedra. That took months of work and consulting with a professor at Oxford.

    The real problem with program verification is the C programming language. In C, the compiler has no clue what's going on with arrays, because of the "pointer=array" mistake. You can't even talk about the size of a non-fixed array in the language. This is the cause of most of the buffer overflows in the world. Every day, millions of computers crash and millions are penetrated by hostile code from this single bad design decision.

    That's why I got out of program verification when C replaced Pascal. I used to do [acm.org] this stuff. [animats.com]

    Good program verification systems have been written for Modula 3, Java, C#, and Verilog. For C, though, there just isn't enough information in the source to do it right. Commercial tools exist, but they all have holes in them.

  • by jalapenokitten (1292142) on Monday May 19, 2008 @03:36PM (#23466074)
    I've been using PC-Lint since about 1997, and have found some pretty nasty bugs using it - in most cases well before they had surfaced in the field. Given that, using such tools is (for me at least) a no-brainer. You do however need to a a particular mindset and take the time to learn how to use such tools effectively - how to tune your warning policy to your codebase, and what issues to look out for which could indicate real trouble (e.g. I've seen behavoural bugs caused by variable name scoping issues which lint caught but which at first glance would appear to be "lint noise"). I have a red flag list of issues I watch for in a new codebase, which works rather well in practice. I should add a disclaimer that these days I make most of my living from working with PC-Lint, so I'll acknowledge my interest in the subject up-front before anyone else points it out!
  • Re:In Short, Yes (Score:3, Interesting)

    by synthespian (563437) on Monday May 19, 2008 @04:32PM (#23466878)
    Don't forget Polyspace [mathworks.com], which I personally have never used - but I would love to. Polyspace is a Standard ML [wikipedia.org] (a higher-order functional programming language) success story, because it relies heavily on the SML MLton [mlton.org] compiler. Another thing that makes it a success story is the fact the Mathworks [mathworks.com] (makers of Matlab) bought it.

    The C/C++ version does MISRA-C [misra-c2.com] (the C used in the automotive industry) too.

    There's also a version for Ada [wikipedia.org], of course.

It is surely a great calamity for a human being to have no obsessions. - Robert Bly

Working...