Slashdot Log In
Anatomy of a Runaway Project
Posted by
kdawson
on Tue Jun 17, 2008 12:47 PM
from the off-the-tracks-and-ploughing-up-dirt dept.
from the off-the-tracks-and-ploughing-up-dirt dept.
JCWDenton recommends a piece by Bruce Webster revealing some insights into a failed multi-million-dollar IT project. "The following document is the actual text — carefully redacted — of a memo I wrote some time back after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Text of Article (Score:4, Informative)
Anatomy of a runaway IT project
By bfwebster on Jun 16, 2008 in Main, Management, Project Failure
[Welcome to reddit and FARK visitors!]
The following document is the actual text -- carefully redacted -- of a memo I wrote some time back [i.e., several years ago] after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered.
Note that the "ABC" consultants were a small part of the overall project team and had been brought in relatively late by "BigFirm" in an attempt to get the "FUBAR" project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: "Bob Winsom", like all the other names in the memo as transcribed below, is a pseudonym.]
CONFIDENTIAL MEMORANDUM -- EYES ONLY
Over the past two weeks, I've conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager's] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.
This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion -- particularly those involving [specific IT] technology -- and who are down in the FUBAR trenches every day. QUALITY OF WORK AND EFFORT
ISSUE: Several consultants said -- and the rest pretty much agreed -- that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.
EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didn't have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value '3, a constant named 'ninety_eight'). Builds take all night. App releases don't run acceptably, if at all, in a production environment. Developers check in files that won't even compile.
RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project. PROJECT PLANNING AND EXECUTION
ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on p
Re:Text of Article (Score:5, Interesting)
Parent
Re:Text of Article (Score:4, Informative)
Also, remember that this was a professional memo written to a high-level manager at BigFirm. It wasn't written to be amusing.
Parent
Re:Text of Article (Score:5, Insightful)
I guess I should say that's what used to bug me about the site; I stopped reading it because of the lies. Look, I understand that it's supposed to be funny, but can't it be true, also? I mean, especially if it's represented as being true. And I can understand a little exaggeration, but some of the changes I've seen between the submission and the published copy are material and, really, unnecessary.
Parent
Re:Text of Article (Score:5, Interesting)
Damn. That's the exact phrase I've been looking for. I don't know how many times I've seen hard truths and unpleasant realities float up the organization and stop dead about 3 management levels below where someone could do something about it. Just like sonar, the people "listening" above the thermocline will never hear anything occurring below it.
Parent
Themocline of Truth (Score:5, Informative)
Parent
thermocline of truth (Score:5, Insightful)
Parent
Re:thermocline of truth (Score:5, Interesting)
This can happen at all levels of the organization, the truth, more or less, might be an open secret. However, for reasons of internal politics and personalities, they need someone from the outside to be the one to tell them.
I once witnessed senior management bring in Peter Drucker [wikipedia.org], at great expense and inconvenience. Before he takes a gig like this, he basically tells the senior management that he gets unlimited access to anyone in the organization, or he's not interested. Then he starts talking to people, in private. He follows threads of interest. He doesn't spread gossip, and he doesn't make judgements, except those in the final report to his client, who is usually the CEO or Board of Directors. In conversation with people he spends almost all of his time listening. With a strategy like that, you can get to the truth of the matter, even though the truth is often very complex, from the standpoint of organizational politics, anyway.
Later, they replaced the CIO, partly on the basis of his report, which most likely didn't say anything like, "get a new CIO" anywhere. Rather it said something like, "this project has already failed, you just haven't realized it, yet," and "here are the indicators that it has failed," etc.
Watching that particular new CIO come in was fascinating, too. He was one of the best CIOs I've ever met. He spent about two months getting his bearings, using essentially the same techniques, only in a manner that seemed lower profile. He just visited with people at different levels of the organization, in the course of normal business, and without any apparent agenda. He didn't march in as a trumpeted turn around artist or anything like that. He was all business. Once he started making decisions, they were big decisions, and they were almost immediately recognizable as the right decisions -- or at least members of the class of possible right decisions for the given problem domain. It took a little while, but people who had been demoralized on this giant failing project (which had involved nearly the entire IT staff to greater and lesser extents) got back a sense of esprit de corps.
Parent
Re:Text of Article (Score:5, Insightful)
Faced with 140,000 lines of obscure cruft which barely performed an extremely simple task, they had the choice between attempting to maintain the monstrosity or start from scratch and do it right. They started from scratch and did it right, which according to the memo involved cutting out 136,000 lines of useless code and vastly increasing the performance.
And you're calling them out on it? You think they did the wrong thing by eliminating 136,000 lines of bloat and significantly improving the performance? You're part of the reason shit like this happens.
Parent
Re:Text of Article (Score:4, Interesting)
Parent
Interesting line (Score:5, Insightful)
Come on now, COBOL isn't that bad. :P. But seriously Java isn't the language you would use for high performance but rather high portability. That says a lot about how bad the original code was.
Re:Interesting line (Score:5, Insightful)
The bulk of most apps is not a hot path and therefore the language is not as important. Even in the hot paths, algorithms often count more than the language. Once a suitable algorithm is determined, performance-critical things are often best written in other languanges (and if it's really critical, in assembly).
Parent
Re:Interesting line (Score:5, Interesting)
ObOwnExperience: One time, I had to do some maintenance work on a very large piece of badly-written and cruft-ridden code, ended up rewriting large tracts of it, reduced its source size by an order of magnitude and the binary size by three orders of magnitude. Also found some buffer overflow Heisenbugs which the previous maintenance guys had known about but bypassed by padding the object files. There's something... bothersome about corrupting a file in order to make a bug not be visible.
Parent
Java can be performant server side (Score:5, Informative)
That old myth? That hasn't been true for many years now, for server side code anyway (which this was describing). Modern JIT compilers make java as fast, and sometimes faster (since you are optimizing code as it runs and not statically beforehand).
But no language will help you if you lack discipline or the ability to code (both of which seemed lacking in this case).
Parent
Re:Interesting line (Score:4, Informative)
Come on now, COBOL isn't that bad. :P. But seriously Java isn't the language you would use for high performance but rather high portability. That says a lot about how bad the original code was.
Parent
This is about Vista, isn't it? (nt) (Score:5, Funny)
The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.
Sounds like Vista to me...
except for the part about (Score:4, Informative)
Parent
Re:except for the part about (Score:5, Funny)
Parent
I get the impression (Score:5, Funny)
Re:I get the impression (Score:4, Funny)
(and I actually really like Java)
Parent
Re:I get the impression (Score:4, Insightful)
Parent
Sigh... (Score:5, Funny)
I am *so* disappointed with the actual article...
Well first... (Score:3, Funny)
Then there's that person who's "kind of a big deal" and thinks the project is "fierce". That would be the senior administrator.
Oh wait. This isn't Project Runway"?
Ouch -- server problems (Score:3, Informative)
Lack of intellectual honesty is endemic (Score:5, Insightful)
Been there, done that (Score:4, Interesting)
We've all gotten burned on projects that got out of hand, but I often wonder why it happens, over and over. Hubris?
I've seen projects where the requirements document was 1000 pages and growing exponentially. For an email server. I remember one project where it didn't matter if code even compiled: we had to ship what we had, because we couldn't delay any further. I made the mistake of expressing my views to the wrong people on that one, and was told in no uncertain terms to shut up if I wanted to continue working there.
I've seen more than one project fall flat on its face because the original requirements were wrong, like trying to develop PC software for an industry that was 100% Mac.
...laura
Not Root Cause (Score:5, Insightful)
In my experience, it is actually not that common for an experienced team to fail largely on execution problems. Rather, as I like to say (call it Renn's Law if you'd like): "Most failed corporate software projects failed before the kickoff meeting". Usually the signs of failure were there all along, before the project even officially got started.
Here are some of the key things I've seen lead to problems, most of which are not directly related to the core development (design, build, and test) of the project:
- Lack of an identified executive sponsor
- Failure to identify a limited subset of people who are empowered and responsible for articulating the business requirements of the system
- Lack of clarity as to the actual goals to be achieved or the underlying problem to be solved.
- No shared vision of what a successful outcome would look like among the various stakeholders
- Project positioned as an IT-centric solution to a business-centric problem without a corresponding business strategy, process, and change management plan in place.
- Insufficient resources (time, money, people) allocated to the project
- Lack of qualified staff in key roles (data architect, functional lead, etc)
- Poor governance and scope control
I laugh at your puny milions (Score:4, Interesting)
I've seen screw-ups totaling hundreds of millions (that we knew about). Possibly billions if some forensic accountants could be sent in. I've worked on the fixes for a few of these. Some so simple that it took half a dozen people about 6 months to successfully build a replacement.
So, where does the money go? I mean $250 million USD on a f*ck-up? The perpetrators aren't driving Ferraris and Bentleys, so I doubt it was embezzled. In truth, project FUBAR was never actually shut down. It still exists as a server with some documents. This allows the company to depreciate the development costs and not have to take a write-down that Wall Street would notice. There's another reason the project, while brain-dead, has its corpse propped up at board of directors meetings:
This outfit is, among other things, a government contractor. They just lost a big one to a competitor, which they are appealing. Part of the reason for the loss is that gov't inspectors, having become aware of the true scope of woe, selected the better alternative. One (rare) example of the gov't actually looking out for our pocketbooks. But in the end, I suspect they (we, the taxpayers, that is) are destined to lose the appeal, since the project is not officially dead and can't be used as the basis of a negative performance report upon which the bid decision was based.
Re: (Score:3, Insightful)
Re: (Score:3, Funny)
Re:Irony (Score:5, Funny)
Parent
Re:Irony (Score:5, Insightful)
I dunno, for it to be ironic Wine would have to have shared some of those characteristics, but it really doesn't.
In particular, the key problem with FUBAR project appeared to be Mr Bob Winsom, whoever he is, who was clearly not technical or competent but believed he was. Wine is led by Alexandre Julliard, who is every bit as competent and skilled as Linus Torvalds himself, if not moreso, the primary difference being that Linus quite a loud person and AJ is not.
Wine has taken a long time to reach 1.0 (a rather arbitrary line in the sand) because Windows is a huge codebase, which is very difficult to match exactly to the expectations of the apps running on it. At its peak Windows had over 5000 engineers working full time on it, something Wine has never had.
Parent
Re:Irony (Score:5, Funny)
Parent
Re:Irony (Score:5, Funny)
Parent
Re:Irony (Score:5, Funny)
Parent
Re:Irony (Score:4, Insightful)
No doubt Winsom is an idiot, but getting rid of him would be about as effective as capital punishment is in eliminating murder (i.e., not very). The problem appears to be systemic and pervasive, like poverty or police brutality.
How can this problem be solved? I have few ideas and little hope. Gall's book The Systems Bible [wikipedia.org] presents some interesting insights.
The one ray of hope currently is FLOSS, whose projects are often free of this particular sort of nonsense. The big problem, of course, is that there seems to be no good, general way to compensate good people for working on these projects...
Parent
Missing the connection (Score:4, Insightful)
Parent
Re:Irony (Score:4, Interesting)
Wine is actually an example of something extremely rare - a project that looked like it was doomed from the beginning, took millenia to get to the current state, but achieved usefulness anyway. Most of the time it works and when it doesn't, most of the time it's just common bugs, not incompleteness.
Parent
Re: (Score:3, Insightful)
In any case, the project to which Mr Webster refers is clearly Microsoft Windows Vista.
Re:I want names. (Score:5, Funny)
Parent
Re:IT Project Managers (Score:5, Interesting)
This has easily been the #1 reason I have personally witnessed for project failure. I am in the process of witnessing it right now, even, with what seems like a relatively simple project. The suits and supervisors along the way are either not responding to requests for information, or change their request for features.
Parent
Intellectual honesty (Score:4, Insightful)
Of course, a lot of people don't really want to be effective in the real world, not as much as they want other things. They want to feel good. They want everybody to like them. They want a quiet life. They want to keep collecting their paycheck. So they stick their heads in the sand and hum the national anthem.
Not that those are bad things to want. But you can't get them just by always picking them in the short term. Easy years require hard moments.
Parent
Re:IT Project Managers (Score:5, Insightful)
In short, a lack of senior staff means a lack of attention, coaching and oversight. If you have too many juniors, your project is going to take a lot longer to correct "newbie" mistakes, and these mistakes are caught later after they're made as well. Either allow for this extra time, or end up with crappy code.
Sadly, the idea has taken hold with upper management that IT is simply a commodity, and as a result most IT shops have become piss-poor at identifying and nurturing talent. They expect junior developer to become "mediors" automatically after a few years, where in practice they have picked up a ton of bad habits on which they've never been corrected. And I expect the shortage to increase in the future... more and more professional IT staff are starting to look for ways out.
Parent
Re:IT Project Managers (Score:5, Insightful)
It all comes down to the fact that somehow the common business sense that people have in every other area seems to go out the window while they are thinking about IT.
Parent
Re:IT Project Managers (Score:5, Interesting)
Parent
But they must be competent (Score:5, Insightful)
A project manager who doesn't actually have the skills it takes to make a project successful will be as bad or worse than no project manager at all. Hiring/retaining more of them will just multiply the problem.
It is very difficult to interview for and find good project managers. The talent pool is just teeming with people who are not skilled developers, and would to love to have a job that is, essentially, just telling other developers to do their jobs. There is tremendous incentive for people who are not competent to be a project manager (or much of anything else, for that matter) to fight tooth and nail for PM jobs. When you hire such a person, your project usually fails, or if it does succeed it is despite, and not because of, the best efforts of your project manager.
Another problem: the best project manager in the world won't get you results if you disempower him or her. I have seen it happen often that the executives see a project slipping and shift into micromanagement mode. At that point, the project manager just becomes a mouthpiece, and the company has robbed itself of the value of their paid talent. If you can't trust your project manager to tell you when a goal cannot be achieved, or when more time must be allocated to some task that doesn't have obvious functional benefits, or that a deadline must be extended, then you have either hired a lemon or you are involving yourself too much in his job. In either case, your project will suffer because of it.
Ok, I will stop ranting now. The bottom line is...more management doesn't solve problems. The right amount of *competent* and *properly empowered* management does.
Parent
Re:IT Project Managers (Score:5, Interesting)
On a project a few years ago (your typical Fortune 500 $LARGE_COMPANY here), our PM was forced to declare that a large release that had taken 6 months to get to "it's kinda working" was "complete" and shipped to UAT even though system testing was incomplete and all we did was give the end users a pile of steaming shit. But the director of the group got to "meet" his deadline, and therefore get his much-desire performance rating.
Of course fixing bugs once the app is in UAT is four times as difficult as in the integration environment, with the corresponding lag in defect correction time. So UAT went on and on and on... until it was supposed to be the final release date. Said users were hysterical and pissed off, and said director was out of his fucking mind trying to come up with inventive ways to ship said steaming pile of shit to production while blaming someone else for the smell of said pile.
His first executive action was to fire the PM and have him escorted out of the building (he was a contractor like me). His second action was to request that the Offshore Solutions team add four more developers to the already swelled-beyond-comprehension development team in India. You know, throwing bodies at the problem. The next thing was to go to his VP and claim that he had been misled by the PM, who by this time was checking out the classifieds at home and therefore unavailable for comment.
In the end, we all went through the usual death march and shipped the thing about three weeks late. The director got his rating (not his fault you understand) and the users had to deal with the remnants of the steaming pile of shit. I get paid either way so no skin off my ass.
Projects are late (or they fail) because the people who are supposed to be in charge of delivering them have no fucking clue as to how software is developed. Fix that problem, and you'll ship all the software you want on time and under budget.
I'm fortunate to be in a project right now where the guy in charge is a former developer who doesn't require a bonus to pay his mortgage, and all the two PMs do is move little bars on an MS Project file while mercifully leaving me and my team alone to actually write the software. We've released two major versions of the app so far the past two years and are on track to deliver the third version sometime this October. On time and under budget. The secret? Iterative development (SCRUM-like) with heavy user involvement in feature sprints. No waterfall bullshit for me anymore, thank $DEITY.
Parent
Every company on earth uses "deliverables" (Score:4, Informative)
All of them. Phrases like that are highly viral and travel through the world management population in under a month.
The article was so generic, it could have described projects I've seen from companies with under a hundred people to a cast of thousands.
Parent
Re:\.ed already ? (Score:4, Informative)
Parent
Re:Merit? (Score:5, Informative)
I was brought in from the outside specifically to conduct a review and summarize my findings in a memo for one specific person in upper management at BigFirm who was above the FUBAR project and who had grave concerns about it, given that at that point the project was years late and millions over budget, and which showed few signs of making it into production anytime soon.
I was not one of the "coders" -- I was not even a project member -- and I certainly wasn't a "new kid out of college"; I graduated with BSCS in 1978; my first programming languages were 360 assembly, PL/1 and FORTRAN, and by the time I conducted this review, I had personally done professional software engineering (including project management, architecture, and consulting) in a wide range of operating systems and programming languages over quite a few fifferent industries.
The ABC consultants, to a person, were likewise very senior software engineers with many years of hands-on coding experience and well-established track records of successful project delivery.
I'm surprised that an IT manager doesn't know what "very fragile code" means. "Fragile code" means that efforts to modify one section of code -- to fix a bug, add functionality, or improve performance -- frequently results in the code "breaking" elsewhere, usually in multiple places. The opposite of "fragile code" is "robust code".
The memo states clearly that previous architects had left (not "project managers"); the problem was that the FUBAR project manager (with no technical background) kept driving them off and, as the memo notes, fancied himself a software architect.
The syndrome of "kingdom building" through increased head-count has long been a major cause of IT project failure; in this case, there were far too many programmers than the problem actually required.
As for my "amateur" status, I'll simply point here [brucefwebster.com]; is the manager willing to do the same? (Sorry about the server problems; I'm raising hell with my hosting service, given what I pay each month for a dedicated server.)
Parent