Why Software Builds Fail

itwbennett writes: A group of researchers from Google, the Hong Kong University of Science and Technology and the University of Nebraska undertook a study of over 26 million builds by 18,000 Google engineers from November 2012 through July 2013 to better understand what causes software builds to fail and, by extension, to improve developer productivity. And, while Google isn't representative of every developer everywhere, there are a few findings that stand out: Build frequency and developer (in)experience don't affect failure rates, most build errors are dependency-related, and C++ generates more build errors than Java (but they're easier to fix).

Because I'm lazy

[OzPeter]

Half the time when I'm working on any sort of non-trivial program (that is too large to hold in my head all at once) and I need to make a breaking code change (and one that is not easily managed with refactoring tools), I'll make the change where it is obvious to me and then let the compiler tell me where it broke and hence where I need to make my fixes.

Re:Because I'm lazy

[QilessQi]

I've spent many decades in that Real World. Ignoring compiler warnings and failing to write automated unit tests for edge cases can cause production defects and database corruption crises that will eat many, many more hours of productivity than simply addressing all compiler warnings. Not to mention causing poor end-user perception and increasing the workload up and down the software support and delivery chain.

Developers whose coding habits cause such situations in real world enterprise or commerce systems are ultimately "less productive" than having no developer at all. :-)

Re:Because I'm lazy

[Imagix]

That one's quite useful. You've declared a variable and now whomever is reading the code now has the additional cognitive load to try to figure out why that variable exists.

Re:Because I'm lazy

[chis101]

If you are talking about C/C++, the variable is *not* null in either case. If you assigned null to it, then it is null. If you never assigned any value to it, then it is whatever happened to be in memory at that location. It's a pretty good warning to let you know you are using a variable without it being assigned a value.

int* ptr;
if( ptr != NULL )
{
*ptr = 0;
}

This code will at some point crash. Maybe not on the first run, but at some point ptr will not be null, but will not be a pointer to valid memory.

Clarifications

[Afty]

Hi, I'm one of the authors of the paper and an engineer at Google. I wanted to clarify some points that have come up in the comments.

First, we don't believe that failing builds are bad. We wanted to study the typical edit-compile-debug cycle that all developers (at least those writing in compiled languages) use to write code. It's perfectly fine to do something like change the signature of a method, compile, then use the compiler errors to find all places where you need to fix your code. We were interested in what kinds of compile errors people run into, how long it takes them to fix the errors, and how we can help you go from a failed to a successful build more quickly. For example, for one particular class of dependency error, we saw that people were spending too much time fixing it. So we created a tool to automatically fix the error and included the command to run the tool in the error message emitted by the compiler. After that we saw the fix time for that class of error drop significantly.

Second, this work is not related to checking in broken code. The builds we looked at are work-in-progress builds from Google developers working on their projects, so it's code in intermediate states of development, not code that has been checked in. It's possible that broken code may be checked in, but our continuous build system will catch that quickly and force you to fix the problem. So for all intents and purposes, all of the code checked into our depots builds cleanly.

Third, by dependency issues we probably don't mean what you think we mean. Within Google we use a custom build system with a custom build file format. Source code is grouped into build targets, and build targets depend on each other, even across languages. You can assume that code checked into the depot builds successfully, and that generally engineers are editing only code in their project and not in their dependencies. The dependency errors we describe in the paper usually result because someone added a source-code-level dependency without adding a matching dependency in the build file, resulting in a "cannot find symbol" error. For example, in a JUnit test you might write the code:
Assert.assertTrue(foo);
But if you don't add a dependency on JUnit to the build file, then you will get a compile error because the build system doesn't know where to find the Assert class. We would count that as a dependency error.

Finally, at Google there is no distinction between "builds on my machine" and "builds on someone else's machine." Our build system requires that all dependencies be explicitly declared, even environmental dependencies like compiler versions and environment variables, so that a build is reproducible on any machine. This is how we are able to distribute our builds. So it's impossible for code to build on a developer's local machine but not on the continuous build system.

I'm happy to answer further questions if people are interested.