Are There Limits to Software Estimation? 225
Charles Connell submitted this analysis on software estimation, a topic which keeps coming up because it affects so many many programmers. Read this post about J.P. Lewis's earlier piece as well, if you'd like more background information.
Too much thought on one thing... (Score:4, Insightful)
Experience
It looks to me like someone just had too much time on their hands, and decided to say that in a very, very complex manner.
Sheesh.
It's all about people (Score:2, Insightful)
What's truly important is that intuitive feel that people develop over time for what the bottlenecks will be, how their particular organization operates, etc, etc...
You can teach number-crunching, but you can't develop that intuition without experience.
An book worth to read (Score:4, Insightful)
by Steve C McConnell
Estimating software development cost (Score:2, Insightful)
The risk and uncertainty stem IMHO from two factors: the importance and rarity of talent and skill (a really good programmer can work ten times faster and produce a finer result than a 'normal' programmer); secondly, the inventive nature of much software development. When you make something new it's impossible to know what surprises you will get.
The more one works with standard pieces and the less one depends on extremes of talent and skill, the more predictable software development is.
Problems with scheduling (Score:3, Insightful)
I don't want to sound like that programmer who makes excuses for why their project isn't delivered on time ("That other guy was a moron", "Management is horrible", "We didn't have solid requirements") but IMHO, if you want a program delivered on time, pick a good team and then try to estimate the amount of time it will take...then reduce that by 20%. It seems like every project is late by about 25% or more, so if you reduce it initially, perhaps it will be delivered closer to when you really expected it.
--trb
Re:Of course there are limits. (Score:3, Insightful)
Not always the case. You're thinking about, essentially, a 'shrinkwrap' environment. Early/late to market and all that kind of stuff. However, a large amount of code is written in-house for in-house use only. Not software houses, but places like banks, government...basically any large institution with a fair few IT systems.
Your users there aren't developers, and many are more used to work which follows predictable patterns. That's not to say their work isn't hard or even non-technical, it's just that the nature of many tasks tends to be more predictable than that of writing code.
Most code is utter drudgery. It's predictable in an informal manner to a very high degree. These users get used to that. If you then predict three days on something, hit an actual problem and take three weeks - they will hold you to account. Your explanation at that time had better be good.
Cheers,
Ian
Re:Accuracy, and doubling a double (Score:5, Insightful)
A task will expand to fill the time available... That's why your method of estimation always seems just right.
Re:Unless it's a simple project... (Score:2, Insightful)
Fred Brookes said "plan to throw one away." Meaning your first attempt at solving the problem may turn out to be a crock of s**t. Somebody else said "if you plan to throw one away, you'll end up throwing two away."
Re:Unless it's a simple project... (Score:2, Insightful)
I read once about a phenomena a lot of people have observed, namely 'second system syndrome'. This is where the second system would be over-engineered and excessively grand, due to all the problems that had been encountered in the first version. 'We've done the first version, we can handle this, so let's make a proper one now'.
I have seen this - it would point to the third version being about right =).
thenerd.
Effort estimation is Irrelevant (Score:5, Insightful)
Project manager says: "We have to add line item X to the project. What's the effort estimate for that?"
Me: "Twelve weeks."
PM: "But we need it in three weeks."
Me: "No way."
PM: "We have to. Shoot for" (names target date in three weeks).
Me: "Sure."
The due date is fixed, and the software development effort is determined by the available time afterwards.
Programmers are not machines..... yet.... (Score:4, Insightful)
Creativity.
It's something that's hard to be measured. Sadly, programming is not like assembling a car, where it can be broken down into infinitesimally smaller chunks, then added back together to get a whole.
For example: it takes six seconds to put this screw in place, so we'll stop the assembly line for 8 seconds, then the car moves on regardless, under the assumption that the screw was inserted.
Programming is not like that. I know I've stared for an hour at the screen trying to figure out why one line of code wasn't working.
Or sat there for a while trying to figure out how to approach a problem before writing another line of code.
Likening programming to a production line is not good. There's no way to know in advance how many lines of code there are going to be, nor how long each line is going to be. If you knew this, you could add up how long it would take the average person to key in the strokes, and there's your estimate. That doesn't work in software.
For time usage, software needs to be compared to any other creative process as opposed to a mechanical one. How long did it take daVinci to paint the Mona Lisa? An hour? Two? 3 days? Could he have guessed from the outset that it's going to take x amount of time? Probably not. He might have been able to give a ball park based on how fast he's painted similar stuff in the past, but he couldn't nail it down exactly.
Now, granted, as you develop time and experience, your estimations get better. In addition, yor time to completion gets better. (How long do you think it would have taken daVinci to paint a _second_ Mona Lisa? A lot less time than the first one, because he's done one, and he remember how he solved various problems, like how much of each color to mix to make a certain tone.) This is where talent and experience come in.
But until software becomes similar to assembling Lego bricks (which it will, one day, and has in some places), then it's going to be hard to quantitatively determine how long a given project will take. And even if it becomes like Lego stacking, there's still going to be some fudge factor because how to solve the problem has to be solved before solving the problem.
And sometimes you have to tear apart and start over because a brick is out of place, or it's just poorly designed.
3 Big snags in time estimates (Score:5, Insightful)
The big problem is that in real-world applications, x is always changing. I have found that the culprits of this is mostly one of several things:
1) You're not as familiar with the project as you thought you were - or there are some aspects that are familiar, but the unfamiliar ones have ramifications you don't foresee because you're not familiar with them. This adds to both your estimations of F(x) and G(F(x)).
2) Users are dumber than you thought. The difference in mindset between the user and the engineer is real and very significant. There are things that as an engineer ( especially one who is working closely to a piece of code for months on end ) you would never try to do with a particular application, and yet a user who has never seen it before will do out of ignorance or confusion or both. Just when you think you've made something idiot proof - they invent a bigger idiot. This throws off your estimates of G( F(x) ) because you have whole classes of bugs you never thought of as bugs before. Sometimes this requires reworking core components making estimates of F(x) go wrong.
3) The client either doesn't know what (s)he wants, or doesn't know how to explain it, or even that it is necessary to be explained. This is the most frustrating of problems, and can be fatal to entire projects. Often clients don't think of software engineering like real engineering. One cannot ask an architect to redesign a building after its already 3/4 built. But this has happened to me with software projects, and even on occasion prompted me to quit a job in frustration. When this happens, all bets on estimates are off.
Either that or I'm just really lousy at doing time estimates =)
Estimation isn't all that difficult (Score:5, Insightful)
The secret is to base your estimate on a detailed specification. Specify in detail, break down the big task into smaller ones, estimate for each smaller task, add up, add 10% for contingency.
I think the problem is that too many estimates are made on the basis of poor specifications, then you get a shock when you discover a problem you haven't anticipated. So, my top tips:
1) detailed spec agreed with client.
2) breakdown into smaller tasks.
3) estimate for smaller tasks.
4) add up and add 10%.
All this stuff about doubling etc. - what are you people like? If you have to do things like that then perhaps project estimation isn't something you should be doing...
The real problem... (Score:2, Insightful)
I develop web applications in a small town. My boss comes to me and gives me specs on some new project. I look over them and give him a quote, say 40 hours, he then proceeds to laugh and say that the client will never pay that much for the app. So we spend an afternoon looking at what we can cut, trying to reuse code, maybe take out a feature or two here and there and come up with half the quote (20 hours) which I tell my boss we can make unless problems arise.
As with all development, problems arise, the client complains about X feature stuff gets redone, the code ends up being a huge mess and usually takes 1 and a half times the original quote.(60 hours). Yet my boss still doesn't figure it out. Why? Probably because his boss keeps breathing down his neck to cut development times as well.
What's worse is when a sales person or my boss talks to a client and gets them to agree on a list of features and the time it takes to develop before even consulting me. Last month a client wanted a content managment system for a website, discussion forums, polls, etc. Because of certain features it couldn't just be downloaded and I ended up just writing it. The client was charged 25 hours, it took closer to 80.
Anyway its the PHBs that cause the problem
Re:The best way to estimate (Score:2, Insightful)
"People under time pressure don't work better; they just work faster."
And it still applies today.--DeMarco and Lister - Peopleware - 1987
The real limiting factor . . . (Score:5, Insightful)
Now, what good is f? On most software projects, f wouldn't be worth much. Why? Because nobody knows what X is. X is a specification of the work to be done (i.e., software requirements), and most such specifications are woefully incomplete, imprecise, and erroneous.
That's why development processes that are repeatable and emphasize increased formalism allow for better estimates. They provide higher-quality X values, not to mention better approximations of f based on past performance. Therefore, if long-term estimates are important to your business, climb the formalism ladder.
On the other hand, good long-term estimates are often unnecessary. Many business need only to know where the project is now and to be able to change directions with reasonable efficiency when business needs change or realities are better understood. Witness the effectiveness of so-called agile development processes in turbulent business environments.
So, in the end, the only real lesson is to pick your software development (and estimating) process to support your business. Doing it the other way around usually doesn't work.
Re:Of course there are limits. (Score:5, Insightful)
Quick tip: If most of your coding is utter drudgery, you're doing the wrong coding.
The potential drudgery I see tends to come from two sources: 1) users tend to ask for very similar things (e.g., a zillion slightly different reports), and 2) tools that are a poor match to the problem domain.
For the first case, users with similar requests, you give the user control and tools to support that control. So instead of wasting your life writing report after report, write report-generating tools.
For the second case, you gotta buy or build better tools. Or, possibly, learn to use the ones you have better. For example, if you're using an OO language, stop using your copy and paste keys. (Why? When you copy and paste chunks of code, you're saying that two things are very similar. Instead of copy-and-paste, abstract the problem using, say, inheritance or containment or delegation. Copy-and-paste yeilds maintenance nightmares.)
Computers do drudgery without complaining, and they do it much faster than you. Make them do your donkey work!
comments from the original author (Score:1, Insightful)
Connell: "no serious researcher in software engineering is trying to find a guaranteed method for producing estimates of time and effort that are certain to be correct. No one is even trying to find methods that produce estimates guaranteed to be correct within a known error range."
By saying this he can ignore those who claim more than subjective estimation because they are not "serious researchers". Fair enough, but you shouldn't leave with the impression that no such claims have ever been made.
and [references in the original paper]R.e. the CMM, read the quote in the original paper carefully and judge for yourself. It's from the beginning of what I call their "manifesto", a white paper that motivates and explains the CMM. Indeed, it stops just short of saying that they claim to estimate objectively, but it sure leaves that impression if you're reading quickly.
More generally, many people in the field don't hesitate to call what they're doing software engineering, some call their methods "scientific". A reasonable definition of engineering is:
Is software estimation engineering of this sort? No.I think of my paper as putting a stake in the ground at one edge of the field, saying "not beyond here", but the rest of the field is open.
An awareness of that 'stake' will help debunk unsustainable claims (if such continue to arise) and also help assign value to methods that really do work, even if the rely on human experience.
And once more, the original paper does NOT say that software estimation is not possible or not valuable. Claiming that it does is a straw man, or more likely, a reflection of the fact that you didn't read it.
Re:Problems with scheduling (Score:3, Insightful)
Absolutely true. Not only are the people doing the work more likely to give good estimates, but people also work much harder to meet their own estimates rather than somebody else's numbers.
estimate the amount of time it will take...then reduce that by 20%
Absolutely false. This is the worst thing you can do for morale. The programmers will know they are working to bullshit targets. And then when they miss the fake deadlines, they'll be stressed and grumpy for the last 20% of the work, meaning you'll get that last part slower and with poorer quality then you otherwise would have.
algorithmic complexity / human complexity (Score:3, Insightful)
Lewis's proof if based on a mathmatical proof that the Kolmogorov complexity is impossible to predict (without actually solving the problem).
One objection was that for some "Kolmogorov simple" problems it may take a human a long time to find the short solution, and that for some "Kolmogorov complex" problems the long solution may be obvious to a human.
It got me thinking. If we fudge the definitions a bit, Kolmogorov complexity still applies. "Thinking" is just another method or language for processing symbols (thoughts). So the Kolmogorov complexity is the length of the shortest sequence of thoughts required to solve a problem. In the general case it is impossible to predict the length of the shortest sequence without actually solving the problem.
-
Re:Programmers are not machines..... yet.... (Score:2, Insightful)
My example was trying to show that there is no way to map out every single step of the process, like you can building a car. It's just not possible. Therefore it's going to be impossible to get an exact estimate.
I understand that lots of the code is easy to estimate, but it's also likely that there will be parts that require months to get working right, or even a long time just to figure out. And it may only be a small fraction of the code, but it may be the most crucial.
Every creative process has parts that are easily estimated. A sculptor can guess how long it's going to carve his basic shape of his sculture from a brick of basic material. But it's hard to guess exactly how long it's going to take to sculpt the actual lines of the face. That's where the difficulty in modeling a creative process lies.
I'm also not saying it's not important, since time = money, especially in business. But a lot of businesses try to think of code as assembling a car as opposed to creating something, and try to manage it that way too.
It seems to me that the author of the article is trying to find a way to exactly quantify how much a program that does "X" will cost. And I'm saying that the best you're ever going to get is a good estimate. Granted, you can refine the estimate, but it's never going to get closer than a certain percentage of accuracy on average, simply because there are parts of programming that can not be quantified.
It makes me cry when this happens (Score:4, Insightful)
Now of course you tell the manager, "If I spend three weeks on a temporary hack, I'm still going to have to spend another twelve weeks later doing it right."
And they say, "Sure! As soon as this crisis is past."
Of course, as soon as crisis A is done, crisis B is looming. And after B, then C, D, and E. So a lot of 'temporary' code gets written. Eventually, the project is just a big heap of steaming turds with some pretty contact paper covering most of the surface. And then the good programmers catch on and leave; the bad ones spend the rest of the lives sticking on more contact paper.
And the manager, of course, has long since moved on; he met his deadlines, after all, so he must be a good manager. And the person who's now in charge of that group? Well he must be a bad manager, because his team has lots of bugs and never makes deadlines anymore.
It's enough to make me cry.
Software Estimation: Impractical Assumption (Score:3, Insightful)
This whole estimation thing assumes that the project parameters do not change during development, which I have never come close to seeing happening on any of the projects I've been exposed to. Ahh.. to be able to work on a project on a fixed set of parameters..
There are the changes that people can never seem to stop making during product development, and they originate from: marketing, sales, superiors, customers, warehouses and factories, just to name a few.
Of course, there are also the factors that you can't predict ahead of time (and consequently, cannot quantify besides adding a qualitative factor) such as changing: product costs, product availability, product specifications, competition, benchmarking, and tool quality/availability.
The best thing I've found is to keep software simple, sweet and very amiable to changing design and specifications. Software estimation is very much an intuitive feel based on past experience; there are also certain characteristics that you know will throw uncertainty into the schedule. For example, not only do I give my superiors at work a "time estimation", but I also give them an "uncertainty" or "risk factor" that tells them how close I feel my time estimation to be. They can learn a lot when you tell them "4 weeks give or take a couple of days" or "4 weeks if it's feasible to do at all".