Why Software Builds Fail 279
itwbennett writes: A group of researchers from Google, the Hong Kong University of Science and Technology and the University of Nebraska undertook a study of over 26 million builds by 18,000 Google engineers from November 2012 through July 2013 to better understand what causes software builds to fail and, by extension, to improve developer productivity. And, while Google isn't representative of every developer everywhere, there are a few findings that stand out: Build frequency and developer (in)experience don't affect failure rates, most build errors are dependency-related, and C++ generates more build errors than Java (but they're easier to fix).
Because I'm lazy (Score:5, Informative)
Half the time when I'm working on any sort of non-trivial program (that is too large to hold in my head all at once) and I need to make a breaking code change (and one that is not easily managed with refactoring tools), I'll make the change where it is obvious to me and then let the compiler tell me where it broke and hence where I need to make my fixes.
Re: (Score:3)
The most important thing is not to avoid that the build fails but to avoid distributing software packages that can't be built.
However if something can't be built due to a mistake it's often easy to find and correct. The big problems are often not that visible and it can take a while to figure them out.
What really grinds my gears is that people release source code that is possible to build, but so full of compiler warnings that you can't be certain that it's going to work as intended.
Re: (Score:3, Interesting)
When in CS, I had a prof that had one rule that for release (not beta/alpha/dev) code, if the code had even a single warning, it was unshippable unless there was an extremely good reason (which would be part of the README) of why it happened. Yes, this was a PITA, but he was trying to teach something that seems to have been lost.
Re:Because I'm lazy (Score:5, Funny)
He's the reason compiler writes invented pragmas to turn off warnings...
Re: (Score:2)
And turning off compiler pragmas are why software is such an amateur engineering field.
Re: (Score:2)
Why do you assume all warnings are useful?? Some of the compiler warnings are just pedantic and are "noise" such as "variable declared but not used", etc.
There is a balance between no warnings and pedantic warnings, namely the useful ones.
Re: (Score:3)
Generally I recommend leaving most warnings on. But sometimes compiler writers go completely over board.
When you use MSVC you have to do stupid stuff like this
#define _CRT_SECURE_NO_WARNINGS // WIN32:MSVC disable warning C4996: This function or variable may be unsafe.
The following compiler specific header suffices to compile code using without warnings, at highest warning level.
#pragma warning( disable: 4061 ) // enum value is not *explicitly* handled in switch
#pragma warning( disabl
Re:Because I'm lazy (Score:4, Insightful)
Why is one reason why no one should ever use Microsoft's code or tools as exemplars. I've got a theory that the first thing Microsoft has a new intern do is write example code to give to customers, and the second thing the intern is asked to do is read the coding guidelines.
Re: (Score:2)
Why do you assume all warnings are useful?? Some of the compiler warnings are just pedantic and are "noise" such as "variable declared but not used", etc.
There is a balance between no warnings and pedantic warnings, namely the useful ones.
One of the things that's nice about the Eclipse IDE is that you can select the importance of selected messages, all the way from "ignore" to "fatal", depending on shop standards and personal paranoia.
However, the offline builders such as Maven and Ant cannot adopt those preferences, so it's not uncommon for a production build to spit out dozens or hundreds of warnings about things that don't actually matter.
Working with C/C++ I almost never had clean builds, since even if I managed to clean one up, it eithe
why women don't code also a solution (Score:5, Insightful)
I know this is "offtopic" but stay with me and I'll bring it around on-topic...
A big question that people are throwing Billions of dollars & millions of internet comments about is "How can we get more women into programming/coding?"
Ok...b/c our industry is by default very complex, it's not unreasonable that to really drill down to an answer to that question might be fairly complex...the answer can be summarized, sure, but to really get at the problem it involves learning a bit.
Here, in this thread, we find out why...and it affects us **all** not just woman coders, or coders...it affects how the whole company works and the perception of value...witness:
Here we have a central thesis:
"There is a balance between no warnings and pedantic warnings, namely the useful ones."
Parent agrees, and describes how using a **proprietary software** (Eclipse) which adds an **extra abstraction layer** to an already ridiculous process...a process which we all know theoretically should be able to be done on a text editor
the fact that coding, the act of developing, software engineering, the 'real work' has such obtuse solutions, solutions to problems based on...
PEDANTIC choices...overkill...the lack of discretion...there are many reasons for this but that's another rant
it's alienating to new people regardless of gender...the only reason many people work jobs as coders is **for the money**
until we address these fundamental issues, the problems that arise only because some compiler programmer was overly pedantic due to lack of empathy skills will destroy any attempt to get non-traditional types into coding
right now, you basically have to be a bit autistic, or be able to think that way on command, in order to code...part of it is genetic, but part of it is deliberate...you have to train your mind to think in a "code" instruction manner...why would a woman do all this given other options?
the solution to pedantic, tone-deaf coding choices is, of course, a fresh perspective that can help get rid of problems from abstractions...
we need women in coding to help make coding more appealing to women
so, to make this on-topic, I think **more women in coding** is a long-term solution to problems in TFA
Re: (Score:2)
why would a woman do all this given other options?
For the same reason men do...there's a deadline to be met. In my experience there are very good female programmers, and very bad female programmers, and many in between. Same as men. Your generalization has no basis.
Re:Because I'm lazy (Score:4, Informative)
Re: (Score:2)
The person reading the code will see things the compiler can't. Like the variable actually is being used, inside some conditional compilation, or that the variable is just a place holder used for debugging (as stated in the comment), and so forth.
But the compiler's job here is to point out possible defects, it is not the compiler's job to provide value judgement on coding style.
Re: (Score:2)
Re: (Score:3, Informative)
If you are talking about C/C++, the variable is *not* null in either case. If you assigned null to it, then it is null. If you never assigned any value to it, then it is whatever happened to be in memory at that location. It's a pretty good warning to let you know you are using a variable without it being assigned a value.
int* ptr;
if( ptr != NULL )
{
*ptr = 0;
}
This code will at some point crash. Maybe not on the first run, but at some point ptr will not be null, but will not be a pointer to valid m
Re: (Score:2)
Before I start, I am not a coder. Never really wanted to be one, but I do understand the principles.
If "int* ptr;" returns any value that is inconsistent it is a problem IMHO. This command should return the exact same value every time, even if it is (empty or zero or whatever).
But then again, there is probably a valid reason why it is inconsistent, and that is why I hate programming. :-P
Re: (Score:2)
However this does not work either! There's a later version of GCC that will see "a=null" and warn "variable 'a' set but not used".
Similarly, code that does "a=a;" used to work for removing this warning in the past, but newer compilers will claim that 'a' is being used before being initialized.
The correct fix apparently is to do "(void)a".
Re: (Score:2)
Re: (Score:2)
I think there is an attitude at times that everyone writes code from scratch, no one deals with legacy code. So why not add several more warnings in this release of the compiler, if no one is affected? Problem is, that variable is indeed used, only it's inside of an ifdef, so now we have to stick in even more ifdefs to hide the variable declaration.
I can understand the idea that perhaps someone had a typo in a variable name. That's possibly worth pointing out. However this gets in the way of "real" warn
Re: (Score:2)
It is noise while you are writing the function, but yes, I agree, it should be addressed once the algorithm has settled down.
Re: (Score:2)
What about warnings that parameters are unused? There are many legitimate reasons for that and yet GCC complains about it. If the API is written that there must be 3 parameters but you don't need the third one, then the programmer should not be forced to have some workaround for this.
As for unused variables, I have seen many legitimate reasons for this, again requiring workarounds to shut up the compiler when there is nothing wrong with the code. Ie, the variable is indeed used but is used inside an ifde
Re: (Score:2)
Which is fine if you're writing new code as you can fix the warnings right then. But for legacy code to get many thousands of warnings merely by upgrading your compiler, it is incredibly frustrating. Just because there is one blip of data in there does not mean that the rest of it is not noise!
The compiler may as well say "Warning: there's a problem with your code" with every file, because there probably is a problem. But that message does not help you find the problem and it will just annoy everyone. Wh
Re: (Score:2)
And some of those warnings are just ridiculously stupid warnings. But if you use -Wall plus -Wextra then you've got to deal with the lunacy.
Ie, complaints about unused function parameters; so what if they're unused, the parameter must be there because of the API and it's ugly code to fix (adding an attribute, cast the paramter to void somewhere in the function, etc). So it makes sense to be able to turn some of these off.
The other big problem is that these warnings are added in later versions of compilers
Re:Because I'm lazy (Score:5, Insightful)
I'm a fan of warnings as much as the next guy, but there are plenty of times that's not practical. Even if you accept nothing else I say, there are plenty of times where third party code (say, for example, Boost...) has warning-producing stuff in it. Do you fix it and maintain your own branch? Submit it upstream, hope it gets accepted, then wait a month for the new release, then demand everyone upgrade to the latest bleeding-edge? That's often (maybe usually) not feasible, which means it should probably just disable it. Fortunately, GCC finally got around to adding #pragmas that sometimes let you disable and re-enable warnings for just the offending headers.
But beyond that, there's also a reason that compilers have most warnings off by default. And why -Wall doesn't turn on anything close to all warnings. And why even -Wall -Wextra isn't all warnings. Because there's a gradation of false positive/false negative tradeoffs, and what's appropriate for you isn't appropriate for everyone. Do you really compile with all warnings, or do you suppress some? Because I can almost guarantee it's the latter, even if you're suppressing them through inaction.
Re: (Score:2)
Re: (Score:3)
Re: (Score:3, Interesting)
The problem is that sometimes that's not an option. For instance, a few weeks ago I was working with some code in VS 2010 that used named enums. Even though Intellisense was smart enough to recognize the enum without the explicit name ("enum" instead of "name::enum"), the compiler kept throwing "unknown symbol" errors if the enum was left as-is, and it would throw a warning indicating that the syntax given was only valid under C++11 if I explicitly scoped the en
Re: (Score:2)
Sometimes the compiler is just being a pedantic bastard. Especially the newer versions of GCC. I honestly think that -Wextra was not intended to be used in practice since so many of those are all highly subjective opinions.
We've got over 2000 source files in our product, not counting header files, so just upgrading the compiler will generate a massive amounts of new warnings. Which means that only the person who's advocating for upgrading away from the archaic compiler is the one forced to modify all tho
Re: (Score:2)
I prefer the warnings and use -Werror for my code.
However, adding -Werror to a library/whatever that the you don't [intend to] control/maintain that has lots of "benign" warnings just causes the headaches that you suggest. But, it leaves the code fragile/open to a bug that the compiler could help with.
But, it's the upstream developer's responsibility to fix the warnings which usually involve less hardship than not fixing them. You never know when it's trivial vs. uncovering a genuine bug. If all the triv
Re: (Score:2)
Clang warns about bad variable names? I need to switch!
Re: (Score:2)
Often with third part products you can't submit it upstream easily, or integrate new changes. Sure that's fine if you're doing open source and have lots of volunteers, but when you have to ship your product in a month you can't waste time integrating newer releases of third party code. So people routinely suppress warnings for third party code.
Re: (Score:3)
When in CS, I had a prof that had one rule that for release (not beta/alpha/dev) code, if the code had even a single warning, it was unshippable unless there was an extremely good reason (which would be part of the README) of why it happened. Yes, this was a PITA, but he was trying to teach something that seems to have been lost.
You should be compiling with warnings as errors as soon as you start coding, and you should fix each one as they occur before you move on to write the next line of code.
Putting of
Re:Because I'm lazy (Score:4, Insightful)
You should be compiling with warnings as errors as soon as you start coding, and you should fix each one as they occur before you move on to write the next line of code.
Putting off fixing these problems leads to bloated and fragile code and wastes much more time debugging and fixing later.
What you should be doing outside the CS class and in the so-called "Real World" is "being productive". That usually means screw the warnings, it has to be completed ASAP or we'll find someone "more productive" than you are.
Re:Because I'm lazy (Score:5, Informative)
I've spent many decades in that Real World. Ignoring compiler warnings and failing to write automated unit tests for edge cases can cause production defects and database corruption crises that will eat many, many more hours of productivity than simply addressing all compiler warnings. Not to mention causing poor end-user perception and increasing the workload up and down the software support and delivery chain.
Developers whose coding habits cause such situations in real world enterprise or commerce systems are ultimately "less productive" than having no developer at all. :-)
Re: (Score:2)
I've spent many decades in that Real World. Ignoring compiler warnings and failing to write automated unit tests for edge cases can cause production defects and database corruption crises that will eat many, many more hours of productivity than simply addressing all compiler warnings. Not to mention causing poor end-user perception and increasing the workload up and down the software support and delivery chain.
Developers whose coding habits cause such situations in real world enterprise or commerce systems are ultimately "less productive" than having no developer at all. :-)
Tell that to the people who equate the time you spend parked in your chair in their offices with productivity.
They're called "Management" and they know that the longer you take the more you're cheating them of their hard-earned bonuses, er profits.
After all it's a simple job that a child/subminum-wage offshore coder/monkey can easily to in short order. All You Have To Do Is...
Re: (Score:2)
I don't need to tell it to them: I am them. As well as being a developer who is, coincidentally, writing some JUnit tests at this moment for my team's current delivery. Time well spent, seeing at the automated tests I wrote earlier today caught an error in our service layer, and our code freeze (for a high volume public-facing website) is in a matter of days.
If you're
Re: (Score:2)
What you should be doing outside the CS class and in the so-called "Real World" is "being productive". That usually means screw the warnings, it has to be completed ASAP or we'll find someone "more productive" than you are.
I'm having a sense of humour failure at the moment, so apologies if this was not taken in the spirit it was intended but: that sort of attitude tends to get you eaten by the Real World for breakfast.
I've very recently found myself working for a company that has gone in that direction du
Re: (Score:2)
The problem is that the set of warnings grows over time. Code that has been good for years suddenly has warnings because a later compiler release starts to gripe about it. Frustrating when the warning is not actually a warning. And this causes teams to delay upgrading compilers for years, because they need to ship the code more urgently than they need to coddle a compiler with the colic. Another thing that some teams do in this case is start ignoring the warnings, and they're forced to ignore the warning
Re: (Score:2)
Yes, this was a PITA, but he was trying to teach something that seems to have been lost.
Yes, he was teaching you to turn off warnings for certain operations so that when the warning was really significant it wouldn't happen.
Maybe if professor think your code is functional but not elegant maybe you should suggest professor write login page himself? Be crazy AND proud.
There is a reason why they are warnings and not fatal errors.
Re:Because I'm lazy (Score:5, Insightful)
NO, he was teaching engineering practices, and a good one.
People like you is why software is in such a terrible state as an industry.
Re: (Score:3)
People like you are what gives people who say "people like you" a bad name.
Re: (Score:2)
It's not a question of turning off warnings, it's a question of correcting the code to get rid of the warnings.
If you turn off the warnings you turn off the warnings for all occurrences in the code, and that is really a dangerous thing to do. In most cases warnings are harmless but in some cases the warning is an important explanation to why something behaves in an erratic way.
Ignoring compiler warnings is stupid, dangerous and can cause serious problems to become hidden.
Re: (Score:2)
It's not a question of turning off warnings, it's a question of correcting the code to get rid of the warnings.
There are two ways of getting rid of warnings. You can either change (potentially a lot of) code, or you can turn off the warning. The fastest way to get rid of them is turn them off. Then you never have to deal with or explain them again.
If I had a boss that said I had to justify and document every warning in the code I work with, and I couldn't turn them off, I'd never get anything productive done. I'd be spending days pouring over someone else's code, making significant changes, and then have to do it
Re: (Score:2)
And it assume you can see the future and predict what will be really serious.
There are many instances of failure becasue it didn't seem like a big deal at the time.
Re: (Score:2)
Is that better or worse than teaching someone to leave warnings on for those operations and then not noticing when a really significant one appears because it's buried in an avalanche of other output?
Worse, because when the significant one appears the code will be non-functional and there will be no compiler warning to help figure out why. It is harder to remember to go back and change the makefile (or other build script) to turn the warnings back on and then see a large number of irrelevant warnings appear (which you may think are relevant and waste time fixing without actually fixing the broken code). If you inherited the build from someone else (not a good idea in a programming class necessarily) yo
Re: (Score:2)
Aaaaaaah! How hard is it to include "stdio.h" from the get go? "Trivial" warnings are usually trivial to fix--and they should be fixed so that the serious warnings stand out. If you let your code develop with all warnings suppressed, than I can see how you can end up in the position where turning them on leads to hundreds of "insignificant" warnings appearing. Don't let that happen. The compiler writers and designers of the language thought long and hard about what should and should not be a warning--
Re: (Score:2)
Aaaaaaah! How hard is it to include "stdio.h" from the get go?
Who cares? How is it relevant to the issue? That was an example, not an exhaustive list of all possible sources of warnings.
"Trivial" warnings are usually trivial to fix--and they should be fixed so that the serious warnings stand out.
I deal with code that has thousands of trivial warnings, and I can tell you that trying to fix them all is NOT trivial. I'm guessing that many of those warnings come from using a different compiler, but I don't know. I have enough to deal with fixing sloppy code that was written using a compiler that silently ignores some ridiculous parameter values but is now being compiled on one t
Re: (Score:2)
Re: (Score:3)
What really grinds my gears is that people release source code that is possible to build, but so full of compiler warnings that you can't be certain that it's going to work as intended.
Where I work, all builds are run with -Wall -Wextra -Werror. So if you check in code that produces a warning, the compiler turns it into an error, and you broke the build. Which means you get to be the build babysitter until someone else breaks it.
Re: (Score:2)
There are even a few warnings that at least with gcc won't show up unless you use the plain "-W" flag, and even cases where they won't show up at the "-O0" level but only at "-O2". And there are a few that you have to enable explicitly.
Add a run of "splint" and/or cppcheck to make sure that the code is as good as it can be. Then execute the binary under Valgrind to make sure that there are no memory leaks. The remaining errors should be those caused by a bad system design rather than plain coding errors. In
Re: (Score:2)
Your QA/Build process is seriously broken if you don't have one person or one team in charge of the build at all times.
Everyone should be familiar with the build system, and a "you break it, you babysit it" rule means that everyone rotates through the responsibility, while also providing an incentive to double check their work. I got the idea from Joel Splotsky's blog post 12 Steps to Better Code [joelonsoftware.com].
Re: (Score:2)
We do have someone that owns the responsibility for the build system, but we also largely subscribe to the "you break it, you fix it" philosophy, and all of the devs have the tools to see when something breaks on the CB system right then and there. Social pressure among the devs ("Awesome job breaking the build, Bob. Did you even compile it before checking it in?") tends to help keep
Re: (Score:2)
Exactly. I routinely break a build to find errors. Want to refactor something? Remove the code it depends on and try to build. Now fix the compile errors. Much easier than trying to make all the dependent code not need the dependancy and then removing it especially when playing with >500k lines of code as I do routinely (you just can't remember to fix everything reliably even when it turns out you did know the full set of places where it was used).
Re: (Score:2)
Most of the times I break something it is in some build for a platform I forgot to try. As in I think this change only affects one product, I build that product and it seems ok, so I check in the code but then later get a build failure email. There are maybe 15 build combinations where only 5 really make a difference, and I'll actually build only 1 or 2 of them locally.
It's usually a computer problem (Score:2)
We've found that builds fail for three reasons: coding errors, dependency issues, command line argument mistakes. If you're a developer you should check these three things when your builds fail and you'll likely find the issue.
Re:It's usually a computer problem (Score:5, Insightful)
Re:It's usually a computer problem (Score:4, Insightful)
Re:It's usually a computer problem (Score:5, Insightful)
Please, give up the C++ slander.
Like any compiler output, read the first error. If you are a developer of any calibre, having a few pages of errors shouldn't phase you and it's not unique to C++ to generate a few erroneous errors. All it requires is a basic level of competence and if you don't possess that then any programming
language that facilitates you generating anything that compiles is doing noone any favours.
Re: (Score:2)
Re: (Score:2)
c++ is great.
Keep repeating it often enough and people will believe you.
Re: (Score:2)
what? You can't tell a missing semi colon from the error messages? or a misspelled word? you're not very good, are you?
LaTeX never fails (Score:3)
My LaTeX builds rarely fail in MiKTeX. The compiler itself seems to be able to download packages and classes from a common repository (CTAN and its many mirrors).
Here's a concept to prevent this crap - UNIT TESTS (Score:3)
Re: (Score:2)
How does a unit test prevent a build error?
Because you have to build your code to run the unit test. The two are (should be) intimately linked. And you should always be building against the latest official working source from the upstream repo.
Re: (Score:2)
And of course everyone always builds with the same configuration, same compiler, on the same platform.
(We have CI servers in our environment. They break not infrequently. Why? Because someone commits a change that builds fine on Linux, and when MSVC gets ahold of it, it produces a warning that GCC doesn't catch and so the build fails. Or MSVC accepts some piece of code that is not actually legal C++ because it's too loose, so when the Linux buildbots get ahold of it, they complain.)
Re: (Score:2)
And of course everyone always builds with the same configuration, same compiler, on the same platform.
Yes, they should, and it should be scripted so that it's trivial so that there's no excuse for not doing it every single time.
Multiple compilers, multiple OSes and multiple binary architectures should all be used and they should all be available to every developer on the network. There should be enough network, storage and CPU capacity to make the builds quick so that there is no excuse for not doing them
Re: (Score:2)
That's fine for something that's intended to run cross-platform, but not so much for something targeted to a specific operating system - code that's hooking Windows drivers isn't going to fare too well under Linux, for instance. As regards the unit tests, it's trivially easy to have code that runs fine in the unit tests but won't build for production
Re: (Score:2)
code that's hooking Windows drivers isn't going to fare too well under Linux
Of course it will, so long as you run Windows under Linux. All you have to do is have your test process spin up a Windows VM on a Linux machine and hook the driver there.
Re: (Score:2)
Good luck automating a dual boot (Score:2)
Re: (Score:2, Insightful)
Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.
Re: (Score:2)
Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.
It does. You shouldn't be putting in new logic bugs with your deliveries. The code should compile cleanly and you should do due diligence to avoid putting in new bugs.
Bugs are expensive to fix, in terms of debugging time and refactoring of broken code to change additional changes built on top of the broken code.
It's amazing how much real progress you can make if y
Re: (Score:2)
Which is fine if you have the good fortune to have a mirror of the build system on your own machine, but in a lot of situations that isn't the case. Where I work we of course have the raw build scripts that we're expected to run locally before committing, but the CB system we use (Jenkins) occasionally just doesn't behave the same wa
Re: (Score:2)
I don't think you understand the study
I didn't RTFA because I could tell from the summary that it was a lot of nonsense.
or c++
I understand enough about C++ to know to avoid it wherever possible. I freely admit that my brain isn't big enough to fight against C++ with enough perseverance to get it to do anything useful correctly. I'll stick to C and scripting languages thank you very much.
or unit testing.
*cough* Test Driven Development *cough*
To become half way good at C++ you either need to devote you
Re: (Score:2)
Re: (Score:3)
If your parents had unit testing you never would've been born.
I would still have been born, just not so buggy.
Re: (Score:2)
Re: (Score:2)
Simple answer (Score:2)
Complexity
Dependencies? (Score:3, Insightful)
Dependencies just magnify all other problems. If your code depends on nothing then it won't break unless the compiler changes. Unfortunately such programs don't exist because you can never depend on nothing and do anything useful. In reality if you depended on nothing you'd end up writing your own console, your own I/O, pretty much your own CRT. This sounds great until you realize your dependency is now the hardware itself and it's likely your code won't be portable in any useful sense. That's why we have kernels.
The problem with C++ is that dependency management is usually file-level and developers 'rarely' care about any file-level constructs (and nor should they, it's an abstract packaging concept). As a result you try to drag in one enum and end up with 100 #includes and 500 new classes you don't care about. This causes bigger object files to be emitted, vastly slower linkage and lots of dependencies you don't expect. All it takes now is for one of those includes to #define something unexpected and BOOM...the house of cards comes crashing down.
Also, did I mention? The C preprocessor causes a lot of grief when it's abused.
Dependencies Problems = "It builds on my machine" (Score:5, Insightful)
Once code is checked in and goes through the standard build process, that's where this is expected to occur because in my experience it's the local environment where the developer does the coding that's the root problem. Why? Developers don't refresh their build environment because of the potential for other problems it may create. I had one gig to unfuck some code at a company a couple of years ago and found out that in order to set up a Dev environment in this place could take two weeks or more depending on what team you were on. You had to go through a script, download this, install that, change this.. A nightmare. Updating dependencies on a local desktop created panics amongst the developers who were reluctant to ever change anything they had which "was working" because you could spend days trying to fix what was broken. Naturally any time they migrated code into test or production (there was no build system) things failed there because of dependency related issues. Also depending on who the developer was, they naturally felt that bypassing the Test/QA cycle was a job perk.
I found dozens of dependencies on desktops that were out of date, deprecated or had major vulnerabilities and that went for the production systems as well. It was bad all the way around from a best practices perspective. Daily production crashes were the norm, the VP of Dev had a monitor on his desk so he could "troubleshoot" production problems it was that bad.
Yes there's shops like this that are still out there.
Re: (Score:3)
I had an experience which was somewhat opposite (though, in a lot of ways pretty much the same).
At one point, the company went with a big giant universal build system.
Every piece of software, every module, every final build ... was recompiled from scratch on a nightly basis. It took a massive server farm many hours to do this. Even if no changes had been made.
What would happen would be someone would break a component. The build of that component, and every downstream dependency broke. The system had no
Re: (Score:2)
This, this, this. Dependency issues on a build server are so often things like "we added this dependency on a library locally, but didn't update the build server", or even "we checked in a change that makes it work on the dev box, but didn't make a change needed for the server". That plus forgetting to merge in stuff is really the only way I ever see build breaks (where checked-in code fails to compile, as opposed to compiles failing locally).
Eighteen THOUSAND engineers?! (Score:2)
Maybe that's part of the problem. Too many cooks spoiling the broth? Perhaps I'm naive, but 18k seems a bit much for what they produce.
Re: (Score:3)
Well if they won't make the beta programs today that will get discontinued tomorrow, who will?
Re: (Score:2)
At all of Google?
Perhaps that'd be true if they were all trying to make the same pot of soup, but in this case it's more like all the cooks in the city with each group in their own kitchen sometimes serving entirely different ecosystems of consumer.
Re: (Score:2)
they produce a lot of stuff. I mean 1000's of things.
I'm not sure you know what they produce.
Hands down, Simplicity of dependencies (Score:2)
So, for instance I like the Crypto++ library because I can cheat and just slam it into my multi-platform codebase without
I blame... (Score:2)
Build failure is just the beginning. (Score:2)
Then comes the installation and packaging failures. The dynamic libs get out of synch, wrong dll gets packaged in, etc
Then the build is good, it does not crash on every pro
Dependencies suck (Score:2)
"Similarly, almost 53% of all C++ build errors were classified as dependency-related. The most common such errors were using an undeclared identifier and missing class variables."
OK, why can't a compiler just Google for the dependency?
Identify everything with a unique ID. When you need to reference it, your system searches all your code (and any other code repositories) for the unique ID.
Why hasn't this been solved?
regression tests and compiling after rewrites (Score:2)
People just forget to do these things, then wonder why the nightly breaks.
Builds fail due to engineer error.
Wouldn't this be expected? (Score:2)
Clarifications (Score:5, Informative)
Hi, I'm one of the authors of the paper and an engineer at Google. I wanted to clarify some points that have come up in the comments.
First, we don't believe that failing builds are bad. We wanted to study the typical edit-compile-debug cycle that all developers (at least those writing in compiled languages) use to write code. It's perfectly fine to do something like change the signature of a method, compile, then use the compiler errors to find all places where you need to fix your code. We were interested in what kinds of compile errors people run into, how long it takes them to fix the errors, and how we can help you go from a failed to a successful build more quickly. For example, for one particular class of dependency error, we saw that people were spending too much time fixing it. So we created a tool to automatically fix the error and included the command to run the tool in the error message emitted by the compiler. After that we saw the fix time for that class of error drop significantly.
Second, this work is not related to checking in broken code. The builds we looked at are work-in-progress builds from Google developers working on their projects, so it's code in intermediate states of development, not code that has been checked in. It's possible that broken code may be checked in, but our continuous build system will catch that quickly and force you to fix the problem. So for all intents and purposes, all of the code checked into our depots builds cleanly.
Third, by dependency issues we probably don't mean what you think we mean. Within Google we use a custom build system with a custom build file format. Source code is grouped into build targets, and build targets depend on each other, even across languages. You can assume that code checked into the depot builds successfully, and that generally engineers are editing only code in their project and not in their dependencies. The dependency errors we describe in the paper usually result because someone added a source-code-level dependency without adding a matching dependency in the build file, resulting in a "cannot find symbol" error. For example, in a JUnit test you might write the code:
Assert.assertTrue(foo);
But if you don't add a dependency on JUnit to the build file, then you will get a compile error because the build system doesn't know where to find the Assert class. We would count that as a dependency error.
Finally, at Google there is no distinction between "builds on my machine" and "builds on someone else's machine." Our build system requires that all dependencies be explicitly declared, even environmental dependencies like compiler versions and environment variables, so that a build is reproducible on any machine. This is how we are able to distribute our builds. So it's impossible for code to build on a developer's local machine but not on the continuous build system.
I'm happy to answer further questions if people are interested.
What is the most common cause of builds failing? (Score:2)
Developers checking in code that won't compile. There case closed.
Re: (Score:2)
Of course, said code module can change, so even if your deps are right now... it might be that the black box code library that does some essential functions might not work the same after it gets updated. In some ways, this is easy to find (if you do a library upgrade and things break, with no other changes.) However, if there are other confounding variables, this might be a fairly difficult task.
Re: (Score:3)
Is this some sort of cry for help? Are you ok? Do you need us to call someone?
Re: (Score:2)
This. I'm inherently suspicious if it actually compiles the first time if I make any sort of significant change.
Re: (Score:2)
Static linking isn't unique to Go. It's telling that you just discovered it, and further more don't realize what it is that you've discovered.
Re: (Score:2)
A build system can help you modularize your build so that a complete recompile does not occur after a syntax or linker error.
For C++? Are you nuts?!