Anatomy of a Runaway Project

JCWDenton recommends a piece by Bruce Webster revealing some insights into a failed multi-million-dollar IT project. "The following document is the actual text — carefully redacted — of a memo I wrote some time back after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered."

Re:Irony

[smeat]

Other than the fact they are delivering something...

Re:Irony

[IamTheRealMike]

I dunno, for it to be ironic Wine would have to have shared some of those characteristics, but it really doesn't.

In particular, the key problem with FUBAR project appeared to be Mr Bob Winsom, whoever he is, who was clearly not technical or competent but believed he was. Wine is led by Alexandre Julliard, who is every bit as competent and skilled as Linus Torvalds himself, if not moreso, the primary difference being that Linus quite a loud person and AJ is not.

Wine has taken a long time to reach 1.0 (a rather arbitrary line in the sand) because Windows is a huge codebase, which is very difficult to match exactly to the expectations of the apps running on it. At its peak Windows had over 5000 engineers working full time on it, something Wine has never had.

Interesting line

[UnknowingFool]

Two consultants rewrote the 140,000 lines of [original obscure language] into 4200 lines of Java. The Java version runs as fast on a laptop PC as the original version runs on a high-powered UNIX server.

Come on now, COBOL isn't that bad. :P. But seriously Java isn't the language you would use for high performance but rather high portability. That says a lot about how bad the original code was.

Re:Interesting line

[Chirs]

Depending on where the bottlenecks are, Java could do reasonably well. (And I say this as a professional kernel developer that works mostly in C, assembly, and shell scripting.)

The bulk of most apps is not a hot path and therefore the language is not as important. Even in the hot paths, algorithms often count more than the language. Once a suitable algorithm is determined, performance-critical things are often best written in other languanges (and if it's really critical, in assembly).

Re:Irony

[Dystopian Rebel]

Anyone else find it ironic that this story about runaway development projects came right after the story on the release of wine 1.0?

Sir, it is not ironic at all, unless Alanis Morissette is your English tutor.

In any case, the project to which Mr Webster refers is clearly Microsoft Windows Vista.

SOUNDS like the typical "mythical man-month"

[zazenation]

approach to project development. "If we hire twice as many programmers, we'll finish it in half the time!"

I ran into this career driven mid-level manager problem-solving approach regularly in the 90's before many of them vaporized (remember DEC?) Time has not changed human nature or incompetent managers.

The PM's of these projects tended to be big on contrived dog-and-pony shows too as I recall.

Re:IT Project Managers

[JaredOfEuropa]

I view these problems as a direct result in regards to a lack of IT project managers.

I find that there's a rather shocking lack of senior, competent technical personnel in general. On a lot of larger projects, there's not a great deal of senior devs to go around so a couple of them end up as dev lead / team lead even though their managerial skills aren't so great. There's no testers to be had so the developers end up doing the testing, and the user acceptance tests end up poorly written aand poorly facilitated. Junior developers have far too much leeway in writing code because there's not enough seniors to coach them, or even do proper code reviews. Application and infrastructure architects are too busy to give each project the attention it deserves, and as a result performance and scalability are not built into the design, and are often not even tested for before release.

In short, a lack of senior staff means a lack of attention, coaching and oversight. If you have too many juniors, your project is going to take a lot longer to correct "newbie" mistakes, and these mistakes are caught later after they're made as well. Either allow for this extra time, or end up with crappy code.

Sadly, the idea has taken hold with upper management that IT is simply a commodity, and as a result most IT shops have become piss-poor at identifying and nurturing talent. They expect junior developer to become "mediors" automatically after a few years, where in practice they have picked up a ton of bad habits on which they've never been corrected. And I expect the shortage to increase in the future... more and more professional IT staff are starting to look for ways out.

Lack of intellectual honesty is endemic

[analog_line]

From large companies to small, it's the single biggest problem they all have. Decisions makers don't want to hear the truth, no matter how loudly they protest otherwise. Anyone with intellectual honesty that doesn't have a previously won huge level of trust from a decision-maker is almost invariably thought to be lying. They all want to have their cake and eat it too, and they will throw money at anyone that tells them they can. Even after getting burned by the consequences of their decisions, less than half (in my experience) bother to try and learn from the failure. Most of them blame the honest person (if they did nothing and as a result failure happened) or latch on to the next person willing and able to lie even more convincingly than the last guy.

But they must be competent

[Anonymous Coward]

It is unfortunately common to knee-jerk a development problem by adding more management.

A project manager who doesn't actually have the skills it takes to make a project successful will be as bad or worse than no project manager at all. Hiring/retaining more of them will just multiply the problem.

It is very difficult to interview for and find good project managers. The talent pool is just teeming with people who are not skilled developers, and would to love to have a job that is, essentially, just telling other developers to do their jobs. There is tremendous incentive for people who are not competent to be a project manager (or much of anything else, for that matter) to fight tooth and nail for PM jobs. When you hire such a person, your project usually fails, or if it does succeed it is despite, and not because of, the best efforts of your project manager.

Another problem: the best project manager in the world won't get you results if you disempower him or her. I have seen it happen often that the executives see a project slipping and shift into micromanagement mode. At that point, the project manager just becomes a mouthpiece, and the company has robbed itself of the value of their paid talent. If you can't trust your project manager to tell you when a goal cannot be achieved, or when more time must be allocated to some task that doesn't have obvious functional benefits, or that a deadline must be extended, then you have either hired a lemon or you are involving yourself too much in his job. In either case, your project will suffer because of it.

Ok, I will stop ranting now. The bottom line is...more management doesn't solve problems. The right amount of *competent* and *properly empowered* management does.

Re:IT Project Managers

[glgraca]

I think we have too many already. The real problem is either that they don't say what needs to be said because they don't want to make waves (they'll twiddle with their Project schedule and pretend that all will be alright) or that upper management simply won't take no for an answer (and will tell them to twiddle with their Project schedule to make it alright).

Re:I get the impression

[cduffy]

Now, now -- by the time DNF comes out, JVMs will run faster than hand-optimized assembler.

Re:IT Project Managers

[COMON$]

Exactly, all of which cascades from a lack of project management. IT project managers are soooo rare no adays that everyone is scrambling to hire them. A good IT project manager will manage each of the problems you noted above. Sys Admins and Dev Gurus are not Project managers. But they get put in the position of being one constantly because upper management doesn't know the difference. It is a completely different skill set. You wouldn't make a simple accountant a CFO or your HR manager a CEO. Sure there is an aspect of accounting to beinging a CFO and there is an aspect of HR to CEO but those are well defined fields that people know how to sniff out good fits for them. However the Professional IT project manager is such a new concept that general managers think any IT guy can fit the bill.

It all comes down to the fact that somehow the common business sense that people have in every other area seems to go out the window while they are thinking about IT.

Merit?

[JeremyGNJ]

Unfortunately for the author of this memo...it seems to lack credibility.

Almost everything stated is based on opinion. It reeks of "amateur", and would be ripped apart by just about any manager it was given to. Here I will show paraphrased examples of what was written by a managers reaction to reading it:

Memo: It has grown too complex to ever go to production
Manager: Please outline the facets of this project that can be eliminated.

Memo: Some coder on my team found a problem with every page of code he sampled
Manager: What makes this coder more qualified than the coder who wrote it?

Memo: The code base is "very fragile"
Manager: What the fuck does that mean?

Memo: My guys took 140,000 lines of old crappy code and replaced it with 4200 lines of Java
Manager: This guy is one of those "I can do it better using the new language i learned in college" kids

Memo: Many previous project managers have left or not been given the power to architect it properly
Manager: This guy is obviously in over his head and fears his job.

Memo: This project has grown too big, probably due to policial reasons of some guy wanting a big important project
Manager: And you're certainly not going to ruin it for me. Don't EVER use the phrase "political reasons" in a professional document.

Re:need maximum verbosity

[jellomizer]

Not to suprising. Older Languges didn't have such a rich default library set.
Obj.sort() (using a fast sorting algorithm) vs. a quick to program bubble sort on the object can obtain performance gains with little extra code. You can write a Web Server in 50 lines in Python vs. 1000 lines in C and still missing functionality in C. That is just because Python has a web server object that intern executes the extra code needed it to run. The same from converting say from ADA to Java. You have a richer languge thus you code less.

Re:Merit?

[fruitbane]

Highly-paid consultants with good reputations can be as honest as they want. They've earned the right. They are not employees and can go find a client who And frankly, if money is being poured into a sinking ship, an ethical consultant has an obligation to spell out that a. the ship is sinking and b. you don't understand the details so I have to give it to you in summary form.

Re:Merit?

[rob1980]

The credibility of the memo lies in the fact that the project ultimately did get canceled. Your example manager here is handling criticism by questioning the credentials and credibility of the folks who were more than likely correct in their assessments of the project, which makes him look like he's just puffing out his chest and mindlessly defending a doomed project for fear of losing his own job - instead of, well, actually addressing the criticism.

Re:Irony

[mkcmkc]

In particular, the key problem with FUBAR project appeared to be Mr Bob Winsom, whoever he is, who was clearly not technical or competent but believed he was.

It would be comforting to think that this fiasco and the many others like it could be traced back to a single incompetent (or sociopathic) individual like Mr. Winsom, but after having experienced a number of these train wrecks first hand, I think this is too facile an explanation.

No doubt Winsom is an idiot, but getting rid of him would be about as effective as capital punishment is in eliminating murder (i.e., not very). The problem appears to be systemic and pervasive, like poverty or police brutality.

How can this problem be solved? I have few ideas and little hope. Gall's book The Systems Bible [wikipedia.org] presents some interesting insights.

The one ray of hope currently is FLOSS, whose projects are often free of this particular sort of nonsense. The big problem, of course, is that there seems to be no good, general way to compensate good people for working on these projects...

Intellectual honesty

[dubl-u]

[Intellectual honesty] has easily been the #1 reason I have personally witnessed for project failure.

It's amazing, isn't it? If you want to be effective in the real world, you have to pay attention the real world. We should dress that up in a lot of bullet points and start a new management fad.

Of course, a lot of people don't really want to be effective in the real world, not as much as they want other things. They want to feel good. They want everybody to like them. They want a quiet life. They want to keep collecting their paycheck. So they stick their heads in the sand and hum the national anthem.

Not that those are bad things to want. But you can't get them just by always picking them in the short term. Easy years require hard moments.

Not Root Cause

[Aaron M. Renn]

I read this with interest as I have been involved in large scale IT development projects for various corporations for the better part of 15 years. This memo makes it appear as if the problems in the project were execution related: bad management, poor quality control, bad architecture, performance problems, etc.

In my experience, it is actually not that common for an experienced team to fail largely on execution problems. Rather, as I like to say (call it Renn's Law if you'd like): "Most failed corporate software projects failed before the kickoff meeting". Usually the signs of failure were there all along, before the project even officially got started.

Here are some of the key things I've seen lead to problems, most of which are not directly related to the core development (design, build, and test) of the project:

- Lack of an identified executive sponsor
- Failure to identify a limited subset of people who are empowered and responsible for articulating the business requirements of the system
- Lack of clarity as to the actual goals to be achieved or the underlying problem to be solved.
- No shared vision of what a successful outcome would look like among the various stakeholders
- Project positioned as an IT-centric solution to a business-centric problem without a corresponding business strategy, process, and change management plan in place.
- Insufficient resources (time, money, people) allocated to the project
- Lack of qualified staff in key roles (data architect, functional lead, etc)
- Poor governance and scope control

Re:Not Root Cause

[bfwebster]

Aaron:

Great post. You'll find a less-general version of Renn's Law in The Art of Systems Architecting by Maier and Rechtin, in Appendix A ("Heuristics for systems-level architecting"): "In architecting a new software program all the serious mistakes are made on the first day." (p. 271). I have quoted that many, many times.

I like your list of key problems. I'm currently writing/editing a book (Pitfalls of Modern Software Engineering) and looking for additional contributors. If you're interested, drop me an e-mail ([e-mail]). ..bruce..

Re:Text of Article

[imidan]

The thing that bugs me about Daily WTF is that the editor often takes great liberties with the stories to make them "more interesting." You'll often find the original submitter of the story down in the comments, telling people what really happened before the hyperbole injection that each story gets before it goes up on the front page.

I guess I should say that's what used to bug me about the site; I stopped reading it because of the lies. Look, I understand that it's supposed to be funny, but can't it be true, also? I mean, especially if it's represented as being true. And I can understand a little exaggeration, but some of the changes I've seen between the submission and the published copy are material and, really, unnecessary.

thermocline of truth

[Gary W. Longsine]

Yes, and unfortunately, the typical bureaucracy enforces this thermocline, brutally. People get fired, not for wasting years and tens of millions of dollars through incompetence, nor for hiding the truth, nor for outright lying about the true status of a failing project, but rather more often, for attempting to fix problems by trying to go around a roadblock that is preventing the truth from going up the corporate ladder. When they are not outright fired, they are certainly blacklisted, and get to watch, as their own career stagnates, and the careers of those erecting the roadblocks advances. As a consultant, I've had the opportunity to see this happen to highly competent, dedicated, organizationally loyal people who should have been given million dollar bonuses, and promoted to Director, or CIO. It's quite sad, but I could never recommend to anyone to be the voice of reason on a project thats failing like this, unless they are prepared to lose their job.

Re:thermocline of truth

[Maxo-Texas]

"you are upsetting the contractors" (moved to spreadsheets for 8 months-- then the project went in and failed-- years later management still pissed at him).

"you wrote a program that helped us through a major catastrophe but without permission and it wasn't planned by management" (denied promotion for 3 years until the entire staff went to the new incoming manager and said this guy wasn't getting a promotion and was a single point of failure)

You are correct. That's the blunt of it. They do not want the truth. So don't get upset. Bypass the chain of command at your peril.

The problem is that tighter controls are taking away our ability to secretly fix things. At a publicly held company they are basically down to a line by line audit of our changes now.

Re:Text of Article

[egomaniac]

There seems to be plenty of blame to go around, but I think it's rather disingenuous for consultants to be rewriting code in Java instead of [original obscure language]. The comparison has no value, and I hope they didn't bill their client for the time. Unless they were brought in to do a Java rewrite, and that doesn't appear to be the case, they should have been spending their time working in [original obscure language].

Faced with 140,000 lines of obscure cruft which barely performed an extremely simple task, they had the choice between attempting to maintain the monstrosity or start from scratch and do it right. They started from scratch and did it right, which according to the memo involved cutting out 136,000 lines of useless code and vastly increasing the performance.

And you're calling them out on it? You think they did the wrong thing by eliminating 136,000 lines of bloat and significantly improving the performance? You're part of the reason shit like this happens.

Re:Risky Redaction

[Bigjeff5]

If you RTFA carefully you would have noticed that that memo was an confidential memo from Bruce('s company) to the heads of "BigFirm" about "FUBAR Project" and specifically conversations with "ABC Company". So yes, Bruce's company was involved, but as an outside analyst to assess the truth about the status of the project. They were not the ABC company brought in to salvage the "Fubar Project".

Good try though. And really, this could apply to almost any large failed IT project. There are so many that you'd have a tough time figuring out exactly which one it was.

The fact that he has coined the (totally awesome) phrase "Thermalcline of Truth" to describe the problem is evidence of how pervasive it is.

Re:IT Project Managers

[drsmithy]

This has easily been the #1 reason I have personally witnessed for project failure. I am in the process of witnessing it right now, even, with what seems like a relatively simple project. The suits and supervisors along the way are either not responding to requests for information, or change their request for features.

That's because if they do so, they then become potentially responsible for any fallout. Which is what's know as a "career limiting event".

This is the beauty of moving up the management tree. Officially, your responsibility (and hence "value") increases, but in reality it actually decreases. You are only "responsible" insofar as knowing who to point the finger at.

TFA is a worthless assessment with no measures

[viking80]

TFA describes a project in complete disarray and a failure on all level. Despite this level of failure, TFA is pretty much anecdotal. TFA has no hard metrics or any other solid data to support the anecdotal conclusions. It is very possible it is accurate, but if a project is a complete mess, it should be easy to get good solid metrics that document this.

The fact that the author of TFA has none of this makes his assessment not only worthless, but places his work in the same category as the mess he was assessing.

If you need to slaughter a project, you need to:
a. Have a measurable set of goals for the project
b. Measure the current status against the goals
c. conclude status.

It would suffice to give a list of the different functions with number of showstopping defects for each function together with a description of the most glaring defect in each of the functions. Just pulled out of the bug tracking system verbatim would do. Example of glaring defect: "When the user push the OK button on screen XYZ, the server crashes, and all the data is lost. There is no workaround"

Re:Irony

[Codifex Maximus]

damn_registrars said:
"Anyone else find it ironic that this story about runaway development projects came right after the story on the release of wine 1.0?"

I wouldn't consider Wine itself to be a runaway project; actually, the runaway project is the system it's trying to be compatible with. :P

You gotta give the Wine guys props. They've pulled a proverbial rabbit out of a hat.

Re:Themocline of Truth

[bfwebster]

Actually, truth tends to die above the thermocline. It's the folks in the trenches who usually have the best idea as to what's going on. ..bruce..

Missing the connection

[try_anything]

The one ray of hope currently is FLOSS, whose projects are often free of this particular sort of nonsense. The big problem, of course, is that there seems to be no good, general way to compensate good people for working on these projects...

I have a similar problem. I know this beautiful, pristine beach. The sand is fine and white, the water is clean, and best of all, it's never crowed. There aren't any tourists except the ones who are willing to hike ten miles in. No fat chicks at all! The only problem is that there isn't any infrastructure. But I'll fix that. I'm going to pull some strings and get a freeway built, and a parking lot, and a McDonald's, and then it will be perfect!

Re:IT Project Managers

[MadKeithV]

IT project managers are soooo rare no adays that everyone is scrambling to hire them. A good IT project manager will manage each of the problems you noted above.

Everyone, repeat after me: Project Managers are NOT Managers . They are glorified Excel & chart monkeys that are expected to report to their boss how well it is going without having any authority at all to actually get reliable data, much less influence what happens. They spend endless hours arguing with a client that didn't want to spend 3 minutes actually explaining what they wanted about how the product doesn't deliver the functionality they never asked for. It's a thankless, shitty job, so I stopped doing that and went into consulting instead. As a consultant and senior developer within the company I've won the trust of developers and can actually change things from the inside out. It's a lot more effective than "project managing" a product to success (hint: it's impossible).