Why Software is Hard 409
GoCanes writes "Salon's Scott Rosenberg explains why even small-scale programming projects can take years to complete, one programmer is often better than two, and the meaning of 'Rosenberg's Law.' After almost 50 years, the state of the art is still pretty darn bad. His point is that as long as you're trying to do something that has already been done, then you have an adequate frame of reference to estimate how long it will take/cost. But if software is at all interesting, it's because no one else has done it before."
Ah! The great unknown... (Score:4, Insightful)
"Interesting" to me means something new and/or unknown...mostly. There are exceptions. Treading new ground always requires greater effort. If I cut a my way through virgin jungle then those who follow have a path.
Programmers (Score:5, Insightful)
Software programming in general is hard for 2 reasons:
1. Computers aren't built for interfacing with humans, thus UI us terribly time-consuming.
2. The environments people like to drop an app into can be so bizarre, that rock-solid stability is very difficult to achieve.
It needs more professionalism (Score:5, Insightful)
Whatever testing is done often only tests that the product produces the correct answers when feed the proper input - no account is taken for how the program reacts to incorrect or incomplete data.
Changes are requested faster than they can be implemented and often are not communicated very well.
In short there are systemic failures throughout the whole process, from inception through to delivery. There is no single answer to why software is hard and there won't be until the industry matures and people start to get thrown out of the business for acting unprofessionally
Software is hard AND there's lots of incompetence (Score:5, Insightful)
Incompetent developers tend to make things more complex than necessary. From that point on, under economic pressure, workarounds are needed to get things done. This in turn makes things even more complex than necessary. THAT is what makes writing software hard. The problem is, it is difficult to be aware of the skills that we lack. As such, a lot of programmers with a huge ego don't deserve one.
I'm not into Extreme Programming per se, but I've noticed that if multiple people look at a piece of software, chances of problems going undetected get smaller and smaller. Yes, even if you, a master programmer, show your code to a rookie, the chance of bugs going undetected will reduce. In fact, it will inevitably result in more bugs being detected before rolling them out to customers.
Re:Becuase People don't know what they want! (Score:4, Insightful)
Re:Ah! The great unknown... (Score:5, Insightful)
Respectfully, I have to disagree. Some of my best ideas have come from pondering over a problem. Pondering can be effort. It's not like daydreaming. To think about a problem and apply logic to try and come up with a resolution requires effort in many, if not most, cases.
My theory (Score:0, Insightful)
At the lowest level, there are bugs made by the programmer. There can be misunderstanding also, especially if there is cultural differences in the organisation.
When multiple programmers are involved there are even more points of failure relating to the interface. Does the module behave as expected? Is the documentation accurate?
At the highest level, is the design accurate enough? Do we even know exactly what we have to do? Do we have the resources or is there wishful thinking involved?
And then, because it can take such a long time to write software, specs change during development. There is people turnover, changing customer need, chainge in direction, etc.
The points of failure are incalculable. The thing to do is adhere to best practices and then take the estimated amount of time to develop and multiply by 2 (in some places, experience will show 3).
And most importantly, don't lose any sleep because of this. Don't want to die at 60 of a hard attack because of software development.
Re:Heavy-handed management (Score:3, Insightful)
Re:Good programming is a boundaries problem (Score:3, Insightful)
Most engineering projects (and software projects...hell, any project) are like children. They're never completed and done. Once you give birth to one, you're stuck with it for a long time....
Re:Product managers... (Score:1, Insightful)
Re:Ah! The great unknown... (Score:3, Insightful)
No you didn't. You said, "Ideas are not the product of labor."
Definition of labor according to Merriam-Webster, just the first/primary definition:
Main Entry: 1labor
Pronunciation: 'lA-b&r
Function: noun
Etymology: Middle English, from Anglo-French labur, from Latin labor; perhaps akin to Latin labare to totter, labi to slip -- more at SLEEP
1 a : expenditure of physical or mental effort especially when difficult or compulsory
Please note: "physical or mental effort". I'm not trying to nitpick. Just disagreeing on our definition of the word "labor".
Software is neither "hard" or "easy" (Score:5, Insightful)
I have a large, global project underway. User requirements are done and have been done, and we're turning those requirements into things we can code or deliver ("View a workorder", "Print asset detail", "Group revisions into single document"). Of that, we have 150 odd deliverable items, not to mention all the fit/ finish work we may have to do, and all of this barely touches on reports, security roles for users, etc.
The reason we're going to make our date, despite the 1280 discrete requirements we need to test, is that we've taken the time to look at the requirements from a few different angles and come up with a solid design plan, before even thinking about implementation. Each piece will build on another, really hard parts are identified early, blockers and such are flagged ASAP. We know things will emerge that we didn't expect, but we've got the biggest chunks identified and working together on paper. We have the flows mapped out, exceptions and variations listed, and a user group that has to sign off on every iteration of the incremental build (we're spiraling out functions and features).
The only thing "hard" about all of this is the incessant thinking about the details, and discipline required to focus on the un-fun part of software construction, i.e. the planning and design walkthroughs. The itch to code something already is growing, but delayed gratification means that when the time comes to actually write something, the design will almost certainly lead to a working, if not optimal, solution. We can refactor as we go, but it needs to work completely before it can work efficiently.
I've been following Chandler off and on, somewhat through Spolsky's references to it and some stray links around the web, and sounds like design didn't go deep enough into what it'll really take to build some of the pieces.
-BA
Many "software engineering" books are trash. (Score:1, Insightful)
A large number of the modern books suggest UML as a solution. Anyone who has actually employed UML knows that it's virtually nothing but hype. Yes, a UML class diagram may be somewhat useful when demonstrating how existing code is structured, and a sequence diagram may prove helpful in showing the flow of messages between objects. But it's unsuitable when used for the design of a large-scale system. One you get beyond 10 or so classes, UML diagrams become too complex to work with, and are basically useless.
The same goes for software patterns. Many do make sense on a small scale. But then the designers of systems try too hard to use patterns wherever possible, the application becomes unnecessarily complex, and failure is often the end result.
Fred Brooks is one of the few authors who has made any real contribution. Of course, that's because he actually had years of experience working on some of the largest, most important and most widely used systems of his time. Even years after some of his writings were first published, they remain one of the most useful guides when it comes to software project management and development. So while some authors babble on about UML and patterns, we need to listen to what Brooks said. Mainly, we need to realize that the main factor is the people who are working on the software, more specifically their knowledge and experience. Solid developers will produce solid software.
Once again... (Score:5, Insightful)
Re:Ah! The great unknown... (Score:4, Insightful)
That's what I was doing; and within the context of a specific provided example.
Think about it. It might take some effort."
Okay troll, right. I've put some effort into it and I'm still clueless. Are you talking about "Ideas may come in a flash, or evade forever."? If so, I consider that a partial truism. Ideas also come about from a slow, plodding, methodical effort. Your generalization is half-assed. If you've got a point to make, please do so. You haven't stated how you disagree with my (and the general use) definition of "labor" and you certainly haven't clearly provided your interpretation of the context involved in the "specific provided example".
Re:Ah! The great unknown... (Score:2, Insightful)
And copyright puts in the toll booth.
Re:Ah! The great unknown... (Score:5, Insightful)
I believe his point isn't that you're not doing work, but rather that scheduling pondering is impossible. Otherwise give me a fairly firm estimate of when you will either prove P = NP, or that you can prove that P < NP. Logical deduction isn't precisely the same as "resolving the unknown". One doesn't provide a time table for when the Twin Prime conjecture will be solved. I can apply logic deduction to lots of problems, but I can't necessarily provide a firm estimate of when I'll find the solution to a problem.
Any time you provide an estimate of the time it will take to do anything in "problem solving", you are using statistical conjecture about how long you think it should taken given that you've solved other similar issues. How long will it take me to resolve a logic puzzle. How long will it take to construct a proof to show something? You think logically on those, but you don't provide a schedule. If you tell me, I'm going to give you 30 different distance, rate, time story problems that are geared for a high school freshmen, I can tell you that I'll be done in about an hour. If you tell me that you'd like me to prove Fermat's last theorem without using reference material. I know it's true, and I know that I can't provide a schedule for it. It's highly unlikely that if I took the rest of my life I couldn't do it. Both require deduction and logical thought. One is an entirely different scale then the other.
When working in the unknown, you can't provide a schedule. Otherwise, you'd be working either in the known, or very close to the edge of known.
KirbyRe:Heavy-handed management (Score:1, Insightful)
"one last time"? - I admire your optimism
Re:Ah! The great unknown... (Score:4, Insightful)
I read somewhere that in science fiction writing this is called "The Tooth Fairy Principle". Don't introduce more than one exotic technology or idea. I immediately realised that it applied even more strongly to software development. New areas represent areas of high risk, adding even a few to a project can change the risk from moderate to very high. I've participated in a few projects who broke this principle ... as usual commenting on the risk that this implied only made me sound like a Cassandra when eventually the prediction bore fruit.
However, the major reasons I see for software projects becoming late are: clients repeatedly wanting to change design after the design phase (in one surreal case we had a client change a fundamental design issue 24 hours before going live!), poor resource allocation (a very large subject), management saying yes to unrealistic deadlines, bleeding edge technology (Tooth Fairy Principle - high buzzword compliance).
Re:It needs more professionalism (Score:1, Insightful)
Re:Not to take potshots, but (Score:5, Insightful)
Re:Heavy-handed management (Score:3, Insightful)
You are right on the money. Programmers tend to try put more priority on building re-useable components (tools, modules, etc.) than on actually building the damn product. I know I've been as guilty of that as anyone.
We're all taught that re-useability, modularity and portability are great ideas. But if you look around at many software projects, these principles are often given top priority, and the cart goes squarely in front of the horse. Few people realize that early architecting can be as evil as early optimization.
How often have you heard of a programmer building a widget system instead of building a widget? Creating a render library instead of creating graphics? Creating an engine instead of an application? Even OOP tells us that we must write a class before writing a function (sorry, method). Anyone remember coding in straight C? It worked. And if you wanted to re-use any of it, it wasn't impossible.
You get the feeling that the quest for productivity becomes a drain on productivity, which, in turn, makes the quest for productivity even more attractive. Then we all wonder why writing software is so hard.
Re:Programmers (Score:2, Insightful)
Actually there is only one master reason. So far, there is no mathematical way to prove that a given non trivial software program will actually work as intended. When designing a physical thing, such as a bridge, machine or electrical circuit, there are precise mathematical formulas which can be applied that give a reasonable expectation that the building or machine etc. will perform according to expectations. There are no mathematics that can be applied that will show to even a ballpark accuracy that a given complex software system, such as an OS, will perform as its designers hoped. That is why there are alpha, beta and all sorts of other test versions where users are asked to try out the programs. My theory is that this is so, because software is not physical, but a pure product of mind to which different rules apply. Human minds do not operate nor communicate with the mathematical precision that computers and machines do. This is in fact what makes us human. All software begins in a human mind which operates in a much different manner than computers. Translating one human language to another human language is hard. Translating human thought patterns into computer instructions, which then have to usually interact with other human thought patterns is even harder.
Re:Programmers (Score:2, Insightful)
Professional Estimators (Score:2, Insightful)
Mythical Man Month (Score:5, Insightful)
Why must we rehash this over and over again.
If it were easy, then yes, we would have push-button frameworks that magically created programs. What bothers me more than the ignorance of the "no silver bullet" mindset, are the pushers of programming environments that supposedly will solve all our problems. Bullshit. Corba sucks. EJBs suck. All these things suck. Just write good software and stop looking for the golden chalice.
Re:Programmers (Score:3, Insightful)
The Mythical Man Month has a better principal. One really good programmer is more productive than 10 mediocre programmers. Or something to that effect.
Re:It needs more professionalism (Score:4, Insightful)
Re:Becuase People don't know what they want! (Score:3, Insightful)
Re:It needs more professionalism (Score:3, Insightful)
It is true that software and computers are relatively new compared to bridges. However, because software is immaterial, it is fundamentally different. There are NO fundamental immutable laws of physics that govern software production. Software is a pure product of mind only. That is the primary reason why software engineering isn't really engineering in bridge or machine building sense. Making good software, especially the kind that interacts with people is a art form rather than engineering a bridge.
Re:Why POOR coding is REWARDED!!!??? (Score:4, Insightful)
Sorry to break this to you, but apps today are not faster than the ones they replaced. We used to write more efficient code, and had more spartan interfaces, because we had to. But I don't think Microsoft Office today is noticably faster than the WordPerfect / Lotus 1-2-3 type apps we wrote in assembler under DOS. All that extra CPU power is more than eaten up by layer upon layer of slower and slower software.
All this allegedly to make programmers more productive. I haven't seen that either.
rd