Distributed Compilation 29
BagOBones writes "Tired of waiting for your source to compile? Dreaming of having your own cluster and having something useful to do with it? Well Trolltech might have the answer. Trolltech Teambuilder lets you turn your network into a clustered C/C++ compiler."
I have nothing against the product (Score:4, Insightful)
That said, I've certainly worked on products that needed at least 3 hours to build an entire tree. These builds were done regularly around 3 in the morning so that the daily drop was available to QA first thing in the morning. It wasn't really necessary to farm out the compilation across machines because it wasn't a big deal to maximize speed.
As for developers' machines, it wasn't like every change was accompanied by a full build. You recompile the files that changed and link the object files together. Any smart build system should be able to handle this type of logic. Such a local build would take about 2 or 3 minutes (if that. This time could be made even shorter by using dlls instead of a single binary). I guess this Trolltech system could reduce this wait even further, but I'm not sure I see the point.
When else are you going to refill your coffee?
Re:I have nothing against the product (Score:3, Interesting)
Sure. OTOH, the project I work on has well over a million lines of C++ code in it, about five developers full time, and a pretty tidy OO design is used to model much of our stuff. If you're playing with a new feature, this quite often goes into a base class to add it across a whole range of areas of the code, and changing the wrong header file can cause a good 10-15 minutes of rebuild (the whole project on a fast machine takes ~1hr). It's the sort of thing that's not normally a problem, but if you do it 10 times in a day while trying to fix that nagging bug, and you're shipping within the week so time is of the essence, then this sort of functionality is priceless. I guess the kind of team where I work is (aside from developing under Windoze) exactly the sort of place they're targetting. To our clients paying a significant sum of money per hour for our time, the $750 to save several hours at crunch time is nothing.
(And before anyone mentions it, yes, I'm aware of the Cheshire Cat idiom and such, thanks. They aren't in common use on the project, which dates from several years ago, and retrofitting them would be difficult to do without a lot of risk. It's cheaper and safer to just speed up build resources, by using faster PCs or something like the product in question here.)
Ack... (Score:2, Interesting)
Re:Ack... (Score:2)
Hay if you want to compile KDE from source it could come in handy.
Re:Ack... (Score:1)
Re:Ack... (Score:2, Insightful)
Re:Ack... (Score:2, Interesting)
Re:The Hypocrisy of /. Regarding MS (Score:1)
As for web standards, you're right, who needs 'em? I'm happy with one company telling what to do. It'll work out in the end. They're rich, right?
Why not dmake? (Score:5, Insightful)
There is a shiney new solution to this (Score:3, Interesting)
BTW, is there "lint"-like utilities for any scripting/interpreted languages that detects "suspicious" stuff before running?
This may help one get the best of both worlds.
I personally prefer interpreted languages. They are usually easier to read IMO because there is less formality in the way.
Interpreted languages attempt to reduce problems by making the view clearer, while compiled languages attempt reduce problems by using a fatter air-bag.
It is sort of like chosing between a fighter plane that is highly menueverable by being light, or one that is heavily armored. They are both different strategies to the same goal. The choice often depends on the pilot and training.
Re:There is a shiney new solution to this (Score:2)
I don't know if that's what you have in mind, but at least Python's exec() and eval() allow for namespace definition (i.e. only let the exec/eval'ed code to have access to a portion of the variable namespace.
ccache (Score:4, Interesting)
That way, if someone has already built the file you're about to build, it's just a copy. Even better is if you have a continous build script caching results ahead of time for you.
It has worked pretty well for me so far.
Re:ccache (Score:3, Informative)
distcc [samba.org].
/elmartinos
Re:ccache (Score:1)
Now I have yet another thing to play around and waste time with... :)
Mosix? (Score:2, Interesting)
http://www.mosix.org/
http://openmosix.sourcef
free with PVMGmake (Score:5, Informative)
distcc (Score:2, Informative)
I don't know about this (Score:3, Interesting)
However, what Trolltech is probably doing, is farming out the independant bits of your program that you would link together in the end anyway. That might really help on rebuilding the entire project at once. However, if your program is set up well, you usually only need to recompile one file at a time.
So the times when you could see the biggest speed up in compilation, you will need this least.
Re:I don't know about this (Score:1)
A build for one platform takes 2-3 hours, which I unfortunately have to test.
Re:I don't know about this (Score:2)
Re:I don't know about this (Score:1)
Re:I don't know about this (Score:2)
While I was working over summer, we regularly made changes to a core IDL file (hey, I wasn't the guy who put all the system interfaces in one file!) resulting in a 4-hour coffee-break on a decent PC And this was just the system layer and GUI. To rebuild everything (including embedded code for various modules) would take 12 hours or more.
Sure you may not recompile everything very often - but imagine what the 12 hours (or 4 hours) would be like if you used the 100 developers PCs on that floor alone. Of course, at some point - probably well before 100 - the cost of organising the distributed compile and transferring files around starts to negate the benefits.
How many full builds do *you* make before each release, when it can take months from initial feature freeze to release? How much will it affect your shipping date? (and how many $$ in wages are you paying people for those hour long coffee breaks all the time?:)
Clearcase (Score:1)
Steve.