When Rewriting an App Actually Makes Sense 289
vlangber writes "Joel Spolsky wrote a famous blog post back in 2000 called 'Things You Should Never Do, Part I,' where he wrote the following: '[T]he single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.' Here is a story about a software company that decided to rewrite their application from scratch, and their experiences from that process."
Missing the point (Score:1, Insightful)
Joel was pointing out a case where an application was rewritten IN THE SAME LANGUAGE. These guys rewrote their application to move it to a new language / platform. It's not a REWRITE, it's a PORT.
There are times for a rewrite and port. (Score:4, Insightful)
I've seen rewrites/ports go quite well. Systems that were originally on mainframes and needed or wanted to be moved to cheaper hardware for cost - if it was the proper thing to do (Sometimes you really need the metal).
Another rewrite that went well was a bunch of code that over the decades became so convoluted to be a maintenance nightmare - modify one thing or add on functionality and then break a shit load of other things.
Just do these basica things and it'll work out.
Go back to the specs and start there.
Talk to the stake holders - yep, there will be creep but also feature reduction because there are things that they never used or because it doesn't make sense anymore.
Plan, plan, plan. No cowboy programming and hacking out shit. And document everything.
It can work.
Re:Here's my short list (Score:2, Insightful)
For me readability > length of code.
I always find it funny when developers are struggling to fit their code on as few lines as possible. Like the forgot how to scroll or something.
Still, if it was written in Visual Basic, I'd let you take care of it.
Re:Ugh (Score:3, Insightful)
The original rewrite of MacOS was Copland, which used up Apples development resources for five years before it was cancelled.
http://en.wikipedia.org/wiki/Copland_(operating_system) [wikipedia.org]
Re:Here's my short list (Score:5, Insightful)
Regarding regexes, I've gone both ways: I've had my developers remove regexes where they were trying to use them, and I've had them add regexes where they were trying too hard to avoid them. Regexes aren't the answer, they're a single tool. You need to use them right. And the /x modifier in Perl helps a lot, allowing you to put useful whitespace and comments in the code (and a regex is code as much as any other language).
As for "as few lines as possible" - you need to do it right. When you span your code over 3 pages, that's not readable anymore. I harp on this with my developers all the time: SCOPE. Scope applies to variables, comments, and functions. The less I need to look at to understand what you're doing, the more likely it is I'm going to understand it. A huge 3-page if-elseif-elseif-else is going to be something I'm not going to understand, as I'll have forgotten all the conditionals by the time I get to the end to know really what scenarios are left - sequential access. A concise regex, on the other hand, is something that I can skim over just by moving my eyes - random access. These concerns aren't just valid for storage media (tape vs DVD/HD). Of course, 40 characters of regex special characters with no whitespace (either horizontal or vertical, preferably both) is generally going to overwhelm most readers, and is going stupid in the other direction.
Yes, readability trumps length of code. But sometimes that means to use a regex (or two or three - why make one big cryptic one when multiple simpler regexes can do the job?). And sometimes, that means avoiding them when what you really want to do is better done by another piece of code.
My favourite new-to-regex example recently has been someone trying to pull apart colon-delimited text with a regex. Woops - there are better language constructs for tokenisation, whether that's strtok in C or it's split in Perl (or better, Text::CSV_XS). Got rid of that regex in a hurry.
Re:And why? (Score:4, Insightful)
We do some small custom web apps for clients, and even a few that have the potential to grow into bigger (but not big) products. While rolling out one of them this week, to the first of three clients who have ordered it, I'm already designing a complete re-write of the core of this product. What started out as a small helper app for one client has turned into a PITA to scale up and out for these other clients. Clearly it's better to fix it now than to keep patching & splicing in order to make it work for them
Re:Ugh (Score:4, Insightful)
The original rewrite of MacOS was Copland, which used up Apples development resources for five years before it was cancelled.
Yeah, I remember that, it's what convinced me to stop being a Mac user. I had the INIT or CDEV or whatever that made your MacOS look like Copland. Then they cancelled it and brought out another shitty MacOS, and I ran like hell and never looked back. All the years from System 7 through System 9 were sad, abusive times to be an Apple customer. They kept bringing out inadequate hardware at astounding prices and almost spent all their cachet. Another year or two and they wouldn't have had enough fans left to float OSX.
Re:Missing the point (Score:3, Insightful)
You're assuming that the original design isn't an organically grown mess of code that's grown and mutated over the last 10+ years (this is pretty common on the business world), preferably written in some proprietary and deprecated language (ASP + VBScript is a current classic, most likely pushed to the company as a good "business language" by some MS sales drone). After ten years of "organic" growth of such apps cleaning them up generally takes longer than just rebuilding from scratch in a sane language with proper separation of presentation, business logic and data storage, exporting the database to the new database and calling it a day. Really, I've done this more than once and sometimes these code bases end up so rotten that it's painful to fix minor bugs, stuff that should take thirty minutes ends up taking the better part of a day because the entire codebase is a total mess with include-o-mania run wild (anyone remember that from the PHP3 days as well? lots of if(condition) { include("filename.php"); } crap that almost impossible to follow).
Summary ignores the single most important quote. (Score:1, Insightful)
From the article: "we experienced a lot of the hardship and trouble that he talks about, so we can easily see how many companies could get into trouble during such a process. If the board of directors had known in 2005 that we wouldn't be ready to re-launch in the international market until 2010, I doubt we would have gotten the green light for the project. "
And that says it all. In this case, it worked out; but it was quite possible that during that 5 year window with no releases that a new competitor could have emerged with a few new features, and these guys wouldn't have been able to respond since their code rewrite was still years away from being ready. Everytime a customer said "We'd love to buy your product but your competitors have feature X and you don't", these guys lost a sale since they couldn't counter "well, we have a new release of our coming out in a couple months with new features". Spending five years rewriting code from scratch instead of incrementally improving the existing code and doing a new release every year could quite possible have caused a company to run out of money and go out of business as customers moved on to the competition.
The reason to iteratively refactor your existing code rather than rewriting it from scratch isn't cause it produces a better end result -- it's cause iterative improvements allow you to periodically stop and ship a release every six months or a year (with one or two new features and a slightly cleaner code base) and then continue refactoring again for the next release. But rewriting from scratch means you can't ship a release for (in this case) ***5 years***. And the world may well have forgotten you by then.
Re:Here's my short list (Score:5, Insightful)
You should think about investing in learning tools. Regex is a well documented, well understood, capable feature of all modern languages. How many weeks will you spend debugging / refining your thousands of if/then/else when you could trust years of testing that has been done on regex engines? Your statement reminds me of novices who avoid ?: in favor of if/else because it's 'cryptic'.
Re:Here's my short list (Score:1, Insightful)
It's a pity that compiler therory is so underused. Not applying Computer Science to write systems leads to lower quality software.
Re:Pick opensource (Score:1, Insightful)
Your entire post is equivalent to a language flaming post, you idiot.
Why should they go with an open source stack? If they're MS-oriented folks, they have more in house experience with those products and they'll be easier, faster, and simpler for them to use.
I'd personally have ported to something multi-platform (no, sorry, mono doesn't count), but that's mostly my technical background and the fact that I prefer to be vendor independent. Other people or organizations without my particular history and concerns can, and do, come up with equally valid solutions to things that differ from mine.
Re:Here's my short list (Score:4, Insightful)
I've massaged if-else code into regexes before. Having several if-elses for a piece of data is rickety and (in my opinion) would take more work to rework than a regular expression if the data changed.
Re:Missing the point (Score:5, Insightful)
Whenever I hit my thumb, I blame the stupid hammer as well.
Or, in other words: a fool with a tool is still a fool.
You can use assembly and have decent code, with clear separation of concerns. Or you can have a 4GL programming tool and still make a mess. Which is exactly why some programmers are 10 times more productive than others.
So where I worked we had ASP+VBScript (supplemented with VB6 COM+ modules running with transactional integrity on an Oracle database) and clean modules, separation of concerns and code that we could easily understand and maintain (even the junior programmers had no trouble getting used to it in a few weeks). We built most of the business apps in the last place I worked on such a design. It still works, is very easy to maintain and transfers cleanly to IIS 8 and Windows 2008. All our database code is in a single (small) module, same as the business layers. Presentation layers is a bit more complex but when transferred to .NET you can just get rid of it altogether because .NET takes over that part. Which is exactly what is happening now, ofcourse.
Don't blame the tools for the lack of ability of most programmers.
Re:Don't pay so much attention to Joel Spolsky. (Score:3, Insightful)
Yes, I never understood why so many pay attention to Joel's inflammatory rants.
Partly because he's a fairly good writer, and partly because he seems to offer simple solutions to the larger problems of software, and partly because he's right at least some of the time.
I DO think he's just dead wrong on this issue though. In my experience starting from scratch is something that should be very carefully considered, but rejecting it outright on general principles is wrong. I really don't care much about his bona-fides or how successful he's been. People with big impressive resumes working for some fashional part of the industry writing in the cool language of the day can say some really stupid things.
The problem starts when people start believing someone PURELY on the reputation alone, and turn their brain off. There's a few figures like that that gain a lot of hero status. Joel is one of them, Bernstein is another. A lot of people actually don't like thinking very much, and prefer ANSWERS to hard questions (even if they're the wrong answers). The cult figures in software offer these answers, defend them to the death, and attract people with similar attitudes. It winds up looking like these guys have large followings, when in reality they have minuscule, but extremely dedicated and vocal ones.
Rewrite if existing s/w congeals (Score:3, Insightful)
Often a rushed, under-resourced or under-skilled s/w project will congeal into a large, brittle, solid clot,
which is not extendable without breaking things in mysterious prohibitive to fix ways.
Congealment comes from insufficient or ill-conceived architecture, and/or rushed or ignorant ill-fitting extensions or mods or "fixes",
combined with insufficient continuous re-factoring.
This code may be worth keeping on expensive life-support if there are many existing customers depending on it,
but make no mistake. Your codebase is already dead, even if its heart still beats.
So then, if you still need software with similar but slightly updated or extended functionality, you should rewrite,
and in doing so, make sure you get good architecture, take sufficient time to build each part or layer, evaluate the quality of
all third party libraries or frameworks used (on the "volleyball" principle that the weakest member of the team drops the ball
and determines the team's i.e. the system's quality), use continuous refactoring, with technical-risk based work prioritization
(biggest risks dealt with first, always), document the classes and methods
sufficiently, and include unit tests and/or invariants and pre-postconditions, so that there is a lower probability that
further extensions will start congealing into brittle, excess complexity.
If you can succeed at maintaining that discipline without going bankrupt, then it will have been worth it, because the value
of your new software capital asset will be much greater than previously.
Of course you should have done it right the first time, (and should have had management enlightened enough to let you,)
because it would have been much cheaper to do it carefully once, than the punishing expense of the original crappy
development and maintenance plus the rewrite. There IS a valid argument that by the time you let your s/w congeal into
a complex, brittle clot, you are already too late, and you should pull the plug, shed a tear, and walk away.
This is a very simple question (Score:2, Insightful)
Whether to do The Big Rewrite always boils down to one very simple question: do the expected gains outweigh the expected losses?
Usually, the argument against doing a rewrite boils down to two key points:
Those are certainly valid concerns, and IME it is often true that their impact is underestimated. However, what the doomsayers tend to ignore is all the potential benefits from writing a second version of something from scratch but with the experience gained from doing it once already:
I’m sure some rewrites really are just because a developer wants to write something new instead of working with what is already there, and those are almost always a bad idea IMHO. On the other hand, it can be annoying if someone comes in assuming that this is the only possible motivation for a rewrite, without considering whether there is another justification for the decision.
My $.02 (Score:3, Insightful)
I'm not a real programmer. I used to play with BASIC back 25 years ago or so and I've fooled around with writing C code for microcontrollers a bit lately but my skills are very limited to say the least. None the less, I was able to parse that regular expression without too much trouble. That leads me to believe that it can't be all that hard for someone who codes for a living to understand it. Still, it would be helpful to spend a minute or two with some helpful commenting.
Re:The exceptions Joel should have included (Score:2, Insightful)
The other problem is with scale.
When a company starts a new project or an entrepreneur starts a small business it's sensible to build something quite small and straightforward. It might take off, but it might not, so don't spend more on the tech than you really need.
So, something gets produced that's VERY hacky. And for a short term solution and from a business perspective, that's quite bright.
The problem is that scale it up a little and it starts falling apart. It becomes hard to maintain because the fundamentals are so poor. It's generally much better to spend the time rewriting at this point than living with what you've got.
Re:Sometimes to move forward (Score:2, Insightful)
This comment is disturbing in many ways.
1) There was and is a KDE player with the table paradigm: juk. It works very well. In the 1.x days, I would use it for tagging, and amarok for the listening, now, I do everything in Amarok, because the tagging works now so much better than t used to, whether in the tagging dialogue or in the playlist.
2) Amarok was always about meta-data around the music: similar tracks/recommended artists, WP articles, lyrics, moodbars and whatnot. In the 1.x days it was so, in the 2.x, it is still so. There never was any other reason to use it, except for the sorry excuse that due to KDE3's broken media system, in many case, amarok was at times the only player actually working. This was and is a wrong reason to use a particular player.
3) Current amarok UI is completely customisable, in fact you can have it identical to the 1.x series. This alone makes your comment inane.
4) clearly if command-line playing does the trick for you, then you never actually wanted to use amarok, whether 1 or 2. maybe MPD is more suited to your usage pattern.
5) The usage pattern of many people is precisely yours: put everything in one big list, and then play it at random (sometimes) . Now for some reason, they need to have the whole list, and will not accept only playing at random. If you are not mentally defective and realise that the outcome is the same, amarok dynamic playlists are much, much better, allowing also for biases in favour of artists/tags etc.
6) Amarok is in fact much faster than it ever was in the 1.x days: if you think otherwise, you have a tiny collection of files. Because search is now instant and was not before. If you care about launch time, then you are relaunching your player a significant amount of times: either there is a bug, or you have OCD. This is not a flaw in the player...
7) Integration with last.fm/podcasts/etc. is cool. You may not need/want it, but who are you to deprive me of a feature I like?
Re:Here's my short list (Score:3, Insightful)
Re:Ugh (Score:3, Insightful)
When I cook biscuits from scratch, I don't start with baking mix, or leftovers. I start with flour, and eggs, and water, and milk, and baking powder,
All of which are components you didn't create yourself. You didn't, for instance start out with flour seeds, chickens, cows, hydrogen, oxygen, an acre of farmland and whatever you'd need to create baking powder.
The point of my analogy is of course that much like in cooking, everyone in software development starts with some ready made component. Is taking NextSTEP off the shelf and starting with that really any different than buying eggs at the grocery store?
Re:Here's my short list (Score:3, Insightful)
Your statement reminds me of novices who avoid ?: in favor of if/else because it's 'cryptic'.
It's not about being cryptic. I use if/else and I've been cutting code for decades. if/else and ?/: do the same thing, while one is easier to read while you're scanning through lines of code (it's just English after all). That the novices can understand what's been written is a bonus.
Re:Don't pay so much attention to Joel Spolsky. (Score:2, Insightful)
It's not like this guy runs a company that does anything critical, like avionics software, or anything really big and tightly integrated like Facebook, or financially significant like Chase's banking system, or leading-edge robotics like Boston Dynamics, or cutting-edge manufacturing like HyperMill. No, they just do Windows and Mac desktop apps. That's trailing edge technology at this point.
Yeah, it's not like he's ever done anything significant...
Spolsky started working at Microsoft in 1991[4] as a Program Manager on the Microsoft Excel team, where he designed Excel Basic and drove Microsoft's Visual Basic for Applications strategy.[5] He moved to New York City in 1995 where he worked for Viacom and Juno.
http://en.wikipedia.org/wiki/Joel_Spolsky [wikipedia.org]
Re:Sometimes to move forward (Score:3, Insightful)
Basically, I am questioning the common wisdom the KDE4 and Amarok 2 are failures. I question them mostly because whatever reproach there might be usually is stale. Stale as in: "this was fixed a year ago, now".
But hey, questioning common wisdom on slashdot is good for burning karma. Remember kids: thrashing free projects based on false impressions you got a year ago is fine. Defending them with actual evidence goes against truthiness, and we can't let a fact-based discussion take place.
But this is something I noticed: I have been hanging on slashdot since 1999 (lost my password/login at some point). It used to be that the general knowledgeability of the audience was much higher. As in, you could actually expect the average slashdotter to know some basics of computer science and of having made some contribution to a free project. Or at least be a scientist/engineer of sorts.
Now, it is mostly ranting wanabees and crazy libertarian/ultraconservative mouthpieces that do most of the talking. Not that they are a majority, be who wants to argue with these...
Re:Here's my short list (Score:3, Insightful)
You can step through a large block of code bit by bit with the debugger. You can't do that with a regex.
I think this post really illustrates the gulf between those who fear regex and those who do not.
A lot of coders today simply do not know how to code (or know how to and avoid doing so), they know how to set breakpoints in a debugger looking for large duplo-blocks of code to comment out or alter in predictable ways until the code outputs what they expect from it.
Now if we are lucky, if we are very lucky the coders who do this are forced to at least write regression tests for their changes. But at the foundational level, they don't know what they changed or why it worked, they just banged the code with a hammer until the rattling stopped. Not having to do so again next month is above their payscale.
Once you abandon this dangerous approach and assume that debugging involves actually grokking the code, then regex suddenly becomes the right tool for a wide variety of parsing jobs. While obviously very terse, it is also rich in it's ability to comment, build larger pattern matches out of smaller, atomic, easily tested pattern matches, etc.
Regex requires someone familiar with Regex to debug. Pages of if-then-else require someone with an automated debugger and no hope of seeing the bigger picture to debug. Regex simply wins this battle, so long as the author is careful to comment and build up well labeled match strings atomically. I am not trying to say that a majority of regex slingers do The Right Thing, I am saying that with if-then-else alone it is not possible to do The Right Thing and no amount of debugger-friendliness changes that.