When Rewriting an App Actually Makes Sense 289
vlangber writes "Joel Spolsky wrote a famous blog post back in 2000 called 'Things You Should Never Do, Part I,' where he wrote the following: '[T]he single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.' Here is a story about a software company that decided to rewrite their application from scratch, and their experiences from that process."
Sometimes to move forward (Score:4, Interesting)
Ugh (Score:5, Interesting)
Drupal does indeed brutalize your database (see second link). So looking forward to D7 to clean this up. That alone was sufficient justification to rewrite the application :p
Unfortunately the author goes on to display his ignorance before this is all over: There are also other examples of total rewrites that have been successful. Apple's transition to OS X is a very good example. The classic Mac OS was an operating system with some good ideas, but the core architecture had a lot of problems. Did Apple make a bad decision to start from scratch with OS X? I don't think so. They brought over and improved the best bits from OS 9, and merged it with the solid Darwin core (much like we did). Uh, no, you are totally and completely fucked here. They started with NeXTStep, last updated in 1995; it was itself based on an older version of BSD (4.3?) and an older version of Mach. OSX is not a complete rewrite of anything. Its legacy stretches back into the 1980s, and so does its code.
There are actually a few good reasons (Score:5, Interesting)
The problem is that companies usually rewrite for all the wrong reasons.
A good reason would be the emerge of a new technology that supports your problem much better, to the point where redoing your code from scratch means easier maintainance later. Usually this goes hand in hand with an old technology (the one you used so far) getting abandoned by its maker. A good example would be how I had to maintain a client/server app written in VB6 using DCOM. Not some quick hack, a full blown client/server solution it was never meant to be, that also has to communicate with a SAP interface and a few more nifty things that cried out "please rewrite me". The overhead to maintain it soon turned from insane to nuts and even the tinyest change required a team of four people to work for weeks.
Unfortunately, the reasons why something gets rewritten are usually different. Like at my next gig. A pretty well designed and fairly solid piece of code was thrown out when the original creator was fired and someone with an ego that required its own office took over and insisted we use "his" collection of libraries instead of that "old" libraries. We trashed about 2 manyears of labour down the drain to end up with what we had before.
The exceptions Joel should have included (Score:5, Interesting)
Re:There are actually a few good reasons (Score:3, Interesting)
Often the problem is that there are things that should be rewritten (either whole apps or specific sections of code) but no-one is willing to green-light such a rewrite even though the rewrite will take less time than bolting hack after hack on top of the old system.
Re:Here's my short list (Score:0, Interesting)
> See if someone put in a bunch of cryptic RegEx, I would re-write it as If-then-else statements.
Rewrite it as regexml! http://code.google.com/p/regexml/ [google.com] ;)
Re:Ugh (Score:5, Interesting)
But Apple did rebuild their GUI shell, application APIs, etc. from scratch. Sure the underlying OS is based on legacy code stretching back into, really, the 1960s (BSD itself was, after all, based on AT&T's Unix), but the GUI and application APIs were totally new.
That is totally and completely false [wikipedia.org]. "Cocoa [wikipedia.org] is one of Apple Inc.'s native object-oriented application program environments for the Mac OS X operating system. It is one of five major APIs available for Mac OS X; the others are Carbon, POSIX (for the BSD environment), X11 and Java. [...] Cocoa is the continuation of several frameworks (primarily the App Kit and Foundation Kit) from the NeXTSTEP and OPENSTEP programming environments developed by NeXT in the 1980s and 1990s." "Carbon descends from the Toolbox [wikipedia.org], and as such, is composed of "Managers". Each Manager is a functionally-related API, defining sets of data structures and functions to manipulate them. Managers are often interdependent or layered.
Newer parts of Carbon tend to be much more object-oriented in their conception, most of them based on Core Foundation. Some Managers, such as the HIView Manager (a superset of the Control Manager), are implemented in C++, but Carbon remains a C API." And I should HARDLY need to touch on POSIX, X11, or Java, the former two of which were already functions of NeXTStep, and the latter of which was not invented at Apple.
There is NOTHING in OSX which can reasonably be believed to have been invented (in any sense of the word) for OSX! Period, the end. ALL portions of OSX contain legacy code, from the microkernel to the presentation layer. They didn't go from Display Postscript to Display PDF because it was easy, they did it because they couldn't just throw over Display Postscript and be able to utilize existing GUI code. They didn't throw out the GUI and replace it, they replaced the engine under it, then made it work on that engine. The appearance of the GUI is still customizable, as it was under NeXTStep, by mangling the images it's made of.
Re:Missing the point (Score:5, Interesting)
Well here's a story from the stone knives and bear claws-era (early 80s):
Two programmers were tasked to convert the Atart VCS/2600 game Pitfall 2 to a Commodore=64 and Atari 800 computer. One said, "The Atari console is so primitive that it's easier to recreate the whole game from scratch," and the other said, "No just copy the 6502 code and then modify it for the varying graphics/sound chips." They then went their separate ways.
- The Commodore=64 programmer recreated the whole game from scratch, and produced a slightly-flawed but decent port.
- The Atari 800 programmer simply dumped the code directly, and then modified it. He produced a port that played identical to the original PLUS he had enough time left-over that he added a whole other game (basically Pitfall 3). So Atari 800 purchasers got two games for the price of one.
Reworking is faster than starting over. Even if the design is a complete mess, there's typically SOME modules that can be reused, and that's time saved
Don't pay so much attention to Joel Spolsky. (Score:5, Interesting)
Look. Spolsky runs a dinky little software development firm that sells a little project management program. And it's still a dinky little software development firm after a decade. It's not like this guy runs a company that does anything critical, like avionics software, or anything really big and tightly integrated like Facebook, or financially significant like Chase's banking system, or leading-edge robotics like Boston Dynamics, or cutting-edge manufacturing like HyperMill. No, they just do Windows and Mac desktop apps. That's trailing edge technology at this point.
Some of the better shops don't hesitate to rewrite. Some have systems which intercommunicate by message passing and are designed to be rewritten in sections. (Facebook, works that way internally.) The bigger shops may have one team on the new version and a maintenance team on the old one; a small firm may not be staffed for that.
Re:Sometimes to move forward (Score:5, Interesting)
you must stop looking backward.
And loose all your users. They are in your 'Backwards" direction.
I have filed or triaged well over 1000 bugs for KDE since KDE 4 came out. Forget about the missing features from KDE 3 to KDE 4, they don't care about breaking compatibility from KDE 4.n to KDE 4.n+1. Even Kaddressbook, one of the core KDE apps, had severe functionality loss from KDE 4.3 to KDE 4.4. Amarok, Koffice, and other lose severe core functionality when their "new versions" were actually "technology previews" with dot-oh version numbers.
Re:Why I want to rewrite (Score:5, Interesting)
I want to rewrite my old code at work... But only for one reason: I am a lot better programmer now than 5 years ago. And 5 years ago, I was a lot better than 10 years ago. And in 5 more years, I have no doubt I'll feel the same way.
There's actually one other reason most programmers would like to rewrite what they're working on: They've solved the problem once, and now they understand it.
IMHO, you can't solve a problem properly without solving it twice. Unfortunately, that's just not, in general, tenable in the industry, and so instead we have things like XP, which encourage prototyping and refactoring, which accept that maxim and attempt to allow for it in the process. Unfortunately, *that* requires preeminent design skills, and that's something lacking in your average developer.
Re:Don't pay so much attention to Joel Spolsky. (Score:4, Interesting)
Yes, I never understood why so many pay attention to Joel's inflammatory rants.
I don't even want to start on his company's product (Fogbugz). Seriously, ASP/VBScript translated to PHP? And then inventing a new programming language just for a web app with ability to output in several other languages? Ugh.
Re:Here's my short list (Score:5, Interesting)
My supervisor is (rightfully) wary of regexes, due to their reputation for unreadability. My favorite solution (in Java): Make a bunch of string constants to match various elements, then assemble those strings together into more strings, which are finally assembled into a single real regex. Sure, it takes about 15 lines for a moderately-complicated regex, but readability is superior to using 15 lines of indexof and substr.
And now, to just bring up a topic that I have a personal interest in:
Nobody knows how to write parsers anymore. I've never seen a recent university CS curriculum that covers parsing with respect to different parser constructs for different languages. Sure, the students learn to load up the chosen XML parsing library and pull in XML, and they learn to take text from stdin, but there's seldom any emphasis on what to do with more screwy formats. Maybe, just maybe, they might get exposed to different languages through a compilers class, but generally not how to process different languages outside of lex/yacc. This bothers me greatly.
Re:Sometimes to move forward (Score:2, Interesting)
The new Amarok just blows period. They could have actually done mockups and put it to a vote using a poll on their site, produced two or three demo UIs with only basic functionality implemented, put it to another vote and gone with that. In the end, the devs really just wanted to reinvent the UI because table views are so 2001.
The result is slow as a pig, ugly and more confusing then the old one.
Hell, I hate it so much that I now just use a Konsole instance running MPlayer instead. mplayer album/*/*.flac FTW.
Re:And why? (Score:5, Interesting)
I have been through a similar project - rewriting a solution that did run under OpenVMS using Basic, Java, C++, C and a bunch of DCL scripts (that confusingly enough for DOS persons have the file extension .COM)
Target environment was Linux and language used was Java 1.6.
My experience when rewriting a legacy system that have a crapload of varying solutions that has evolved during 25 years or so you will find that there is always yet another functionality that nobody told you about - effectively doubling the development time. (This "Multiply estimated time by PI factor" statement isn't that far off.)
And there were some traps involved too - migration of the system had to be seamless for the users as much as possible and with minimal downtime. Since there were over 400 different customers with everything from 1 to 1000 users each involved this was to say the least "tricky". Especially since this was a 24x7 system. The solution was to write a replication protocol that replicated data between the old system and the new. The old system used OpenVMS indexed files while the new system runs a MySQL database and the data structures were different too, which made it necessary to write a replication solution. So when a customer was migrated it was effectively done by setting a flag that redirected them from the old system to the new system and they could continue working.
Of course there were bugs in the beginning, and user errors since the new system did have different functionality and behavior compared to the old. Bot none of them were causing any irrecoverable problems. Invoice printing was delayed, but no major amount of money was lost. The majority of the problems appearing didn't affect the end users at the customers, only the helpdesk service personnel and they were prepared for limitations ahead of time.
The amount of downtime for the system during the two years it has been operational has been very low. And this has given a different concern - too few "problems" with a system is also a problem because tech support will almost forget that it exists.
Specific problems with the application - especially in the beginning has been running out of PermGen space in Java. This at least partly due to design mistakes. But memory leaks that grows over time are very low. And the use of FindBugs [sourceforge.net] has been very useful to trap a lot of errors (potential and real).
What the application does? - It's a management application for short-term lease of telephony at hospitals and similar (almost 400) and other services (a few) which enables and disables phone extensions, assigns numbers, allows instant move of an extension and provides invoicing for the rental and phone usage through processing of CDR:s.
Re:Here's my short list (Score:3, Interesting)
Nobody knows how to write parsers anymore. I've never seen a recent university CS curriculum that covers parsing with respect to different parser constructs for different languages. Sure, the students learn to load up the chosen XML parsing library and pull in XML, and they learn to take text from stdin, but there's seldom any emphasis on what to do with more screwy formats. Maybe, just maybe, they might get exposed to different languages through a compilers class, but generally not how to process different languages outside of lex/yacc. This bothers me greatly.
Warning: Long rambling post follows. To summarize, I haven't found any courses that covered anything close to parsing, except maybe invoking XML parsing libraries and stuff.
Before I took any programming courses, I figured out how to create an XML parser in javascript. I had a dream of making a game, which needed server-side storage, but I never got around to finishing it. Quite an experiment, though.
After that I went on to help decode Outpost 2 [wikipedia.org]'s compiled map format. (maps were DLL files - oh the design choices of old games!) I wrote an editor for it in Jamascript (a mirror of Javascript - see Clickteam [clickteam.com]), and wrote my own tile renderer with smooth scrolling. It worked quite well, except in XP SP2. Lack of further interest from the community caused that project to die, but got me interested in compiled languages - with a compiled language, I'd be in control of whether it worked on an OS or not.
So I decided I needed to learn a compiled language, but since I didn't like C, I started by porting my XML parser to Java, and added writing capabilities. I used it for some projects where I didn't need to deal with DOM nodes and stuff; I just needed to spit out data as XML <key>value</key> pairs, which it did just fine. Although the parser was simple(200 lines, max?), and couldn't read complex XML, it could generate valid simple XML, which any other parser could read. Oh, and it failed gracefully if the file was too advanced. (that was my biggest accomplishment, and required rethinking how I had designed it)
Then I changed it to pre-pend itself to other files as a sort of metadata. That way I could attach info like the number of frames, size of frames, etc. to a tileset PNG. You can see where I was going with this... I wanted to make a game. But somehow I got bored and stopped working on it again.
All this working with Java led me to a few inevitable conclusions. I just plain don't like some of Java's syntax verbosity. Some of the choices make the code stretch on and on and on, when readability could be enhanced by doing the opposite. I wrote a code pre-processor, to enable (among other things) keywords like public: and private: - also var, so that I could easily deal with statements like this:
public ArrayList<SomeObject> myList = new ArrayList<SomeObject>();
->
var myList = new ArrayList<SomeObject>();
It just felt cleaner to me - perhaps because of my javascript roots. I had a dream of creating an entire IDE, with background colours (changing based on scope, public/private, etc.), but I never actually got around to writing that... in the end I went back to other IDEs which lacked my features, but had stuff like code completion. (quite handy)
Now I'm working on many other projects - but I must say, I haven't encountered a course yet that covered any of this stuff. I'm probably looking at the wrong college courses, or maybe there's just not enough jobs that demand it? I had no trouble finding, Ruby, Python, Java, C#, and C++ courses, but none cover this stuff. I just had to tinker to learn it.