Slashdot Log In
Why Software is Hard
Posted by
Zonk
on Sat Feb 03, 2007 04:52 PM
from the comedy-and-software-are-in-the-same-club dept.
from the comedy-and-software-are-in-the-same-club dept.
GoCanes writes "Salon's Scott Rosenberg explains why even small-scale programming projects can take years to complete, one programmer is often better than two, and the meaning of 'Rosenberg's Law.' After almost 50 years, the state of the art is still pretty darn bad. His point is that as long as you're trying to do something that has already been done, then you have an adequate frame of reference to estimate how long it will take/cost. But if software is at all interesting, it's because no one else has done it before."
Related Stories
[+]
News: The Death Of CS In Education? 521 comments
JohnnyKimble writes "A provocatively titled article recently appeared in the 'Future of Computing' section of the British Computer Society website. 'The Death Of Computing' was written by a lecturer at De Montfort University in the UK, and considers the problem of falling interest in computer science courses in the UK and what needs to be done to encourage more students to take the courses." This ties in well with our discussion last night about Why Software is Hard.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Not to take potshots, but (Score:5, Funny)
Re:Not to take potshots, but (Score:5, Funny)
Another law explains it, Entropy.
Parent
Re:Not to take potshots, but (Score:5, Funny)
Parent
Re:Not to take potshots, but (Score:5, Funny)
Parent
Re:Not to take potshots, but (Score:5, Insightful)
Parent
Re:Not to take potshots, but (Score:5, Informative)
This is a point I've come to also after 72 years, the last 35 or so of it coding this and that although not much recently as I seem to be fadeing into the dim sunset of SS mentally.
When working in assembly, then it may be that one person is the optimum number of coders as I've done some never before been done stuff in assembly several times in the early years, sometimes with hand assembly (on an 1802 board) where you look up the nemonic and enter the hex equ in a hex monitor. It took me about 6 months to fine tune about 3k of code, but it was still running 12 years later when I last checked in at that station. And still saving the station 2-3 man hours a day and giving them a better air product at the same time.
But for a higher level language, I think 2 can be more productive, particularly when one knows what he wants to do, and the other knows how to do it once its properly outlined. Many times the coder himself is simply too close to the code to see the job it has to do, but the partner in turn has a good idea of what its got to do. The genesis of at least 2 fairly well known amiga programs were from the mind of a younger man in another dept at the tv station, and he would hack up what he thought might work but didn't, but once I knew the requirements, the final code more than likely came from my keyboard. He had the imagination that I lacked, possibly due to my advanceing age, and was in turn concentrating on his job's duties which I wasn't always aware (I had other responsibilities too) were being done at less than optimal methods. We sure made a good combo crew though.
I have NDI how many man-years in in vista right now, but I dare say it is a substantial investment in both time and programmer salaries. I'd also wager that at least 75% of any one programmers day was spent conferring with other programmers as to the best way to do it, and get it done within the generally immutable confines of the
As for re-using known good code ideas, or a 150 line snippet here and there, it is to be encouraged at every staff meeting, re-inventing the wheel is not good use of his time and as others have said, only serve to intro new bugs that then have to be run down and fixed. Programmers really should get over the attitude that I can write it quicker, and spend more time reviewing older code to see if it can be recycled. There is much knowledge in 10 year old code thats still in use everyday.
Will my little treatise make any difference at the end of the week? Donbesilly, This is after all,
--
Cheers, Gene
Parent
Ah! The great unknown... (Score:4, Insightful)
"Interesting" to me means something new and/or unknown...mostly. There are exceptions. Treading new ground always requires greater effort. If I cut a my way through virgin jungle then those who follow have a path.
Re:Ah! The great unknown... (Score:5, Informative)
When you innovate in a game, only make one....maybe two innovations. Otherwise, you skew so far away that you usually end up a complete failure. Applying it here: sure, keep things interesting by doing some piece new, but keep it manageable by keeping the rest of it "boring". You gain predictability while retaining "fun".
Layne
Parent
Re:Ah! The great unknown... (Score:4, Insightful)
I read somewhere that in science fiction writing this is called "The Tooth Fairy Principle". Don't introduce more than one exotic technology or idea. I immediately realised that it applied even more strongly to software development. New areas represent areas of high risk, adding even a few to a project can change the risk from moderate to very high. I've participated in a few projects who broke this principle ... as usual commenting on the risk that this implied only made me sound like a Cassandra when eventually the prediction bore fruit.
However, the major reasons I see for software projects becoming late are: clients repeatedly wanting to change design after the design phase (in one surreal case we had a client change a fundamental design issue 24 hours before going live!), poor resource allocation (a very large subject), management saying yes to unrealistic deadlines, bleeding edge technology (Tooth Fairy Principle - high buzzword compliance).
Parent
Re:Ah! The great unknown... (Score:5, Interesting)
http://www.austincc.edu/techcert/Video_Games.html [austincc.edu]
It's not a degree program (yet), but I'm not too worried about that since I already have a CS degree. For me, it's more about having fun, learning some new stuff, and making good contacts for when I'm ready to jump into the industry.
Check out the list of names on the Advisory Board and the list of Instructors. There are some influential names on that list.
Layne
Parent
Re:Ah! The great unknown... (Score:5, Informative)
In fact, I think that some of my business experience helps me more than others in the program. I have a better feel for structure within a process than most of them. Scope / Requirements / Design / Code / Test translates into the game world as Pitch / Game Design / Technical Design / Code / Test (with Art being like having a Web Tech team that does the HTML and Styles for you).
But if you ever did any old school Windows programming (where you had to actually hand roll your event loops), that's basically the core of game programming. Everything else is event handling (fire event, score event, death event, etc.) and calling libraries (graphics, sound, etc.). Granted, that's boiling it way down, but equating it that way should give you an idea of how easy it really is.
Layne
Parent
Re:Ah! The great unknown... (Score:5, Insightful)
Respectfully, I have to disagree. Some of my best ideas have come from pondering over a problem. Pondering can be effort. It's not like daydreaming. To think about a problem and apply logic to try and come up with a resolution requires effort in many, if not most, cases.
Parent
Re:Ah! The great unknown... (Score:5, Insightful)
I believe his point isn't that you're not doing work, but rather that scheduling pondering is impossible. Otherwise give me a fairly firm estimate of when you will either prove P = NP, or that you can prove that P < NP. Logical deduction isn't precisely the same as "resolving the unknown". One doesn't provide a time table for when the Twin Prime conjecture will be solved. I can apply logic deduction to lots of problems, but I can't necessarily provide a firm estimate of when I'll find the solution to a problem.
Any time you provide an estimate of the time it will take to do anything in "problem solving", you are using statistical conjecture about how long you think it should taken given that you've solved other similar issues. How long will it take me to resolve a logic puzzle. How long will it take to construct a proof to show something? You think logically on those, but you don't provide a schedule. If you tell me, I'm going to give you 30 different distance, rate, time story problems that are geared for a high school freshmen, I can tell you that I'll be done in about an hour. If you tell me that you'd like me to prove Fermat's last theorem without using reference material. I know it's true, and I know that I can't provide a schedule for it. It's highly unlikely that if I took the rest of my life I couldn't do it. Both require deduction and logical thought. One is an entirely different scale then the other.
When working in the unknown, you can't provide a schedule. Otherwise, you'd be working either in the known, or very close to the edge of known.
KirbyParent
Re: (Score:3, Insightful)
No you didn't. You said, "Ideas are not the product of labor."
Definition of labor according to Merriam-Webster, just the first/primary definition:
Main Entry: 1labor
Pronunciation: 'lA-b&r
Function: noun
Etymology: Middle English, from Anglo-French labur, from Latin labor; perhaps akin to Latin labare to totter, labi to slip -- more at SLEEP
1 a : expenditure of physical or mental effort especially when difficult or compulsory
Please note: "physical or mental eff
Re:Ah! The great unknown... (Score:4, Insightful)
That's what I was doing; and within the context of a specific provided example.
Think about it. It might take some effort."
Okay troll, right. I've put some effort into it and I'm still clueless. Are you talking about "Ideas may come in a flash, or evade forever."? If so, I consider that a partial truism. Ideas also come about from a slow, plodding, methodical effort. Your generalization is half-assed. If you've got a point to make, please do so. You haven't stated how you disagree with my (and the general use) definition of "labor" and you certainly haven't clearly provided your interpretation of the context involved in the "specific provided example".
Parent
Programmers (Score:5, Insightful)
Software programming in general is hard for 2 reasons:
1. Computers aren't built for interfacing with humans, thus UI us terribly time-consuming.
2. The environments people like to drop an app into can be so bizarre, that rock-solid stability is very difficult to achieve.
Re:Programmers (Score:5, Funny)
Parent
Re:Programmers (Score:5, Funny)
It is well known that men are superior in the kitchen.
Parent
Re:Programmers (Score:5, Funny)
Parent
Re:Programmers (Score:5, Funny)
Wow... My first ever post that got modded down as flamebait. Awesome :-) Especially funny, considering the parent post which was blatantly sexist got modded up as insightful.
Parent
Re:Programmers (Score:4, Interesting)
Neither condition hold for programming. It's for this reason that I think that, in general, *two* programmers can program faster than one. At least, me and my partner can program code that's more bug-free together than we can when we program separate projects, and that makes a difference. If the project is sufficiently large - i.e. takes longer than about 10 hours, the cost of communication between two people is less than the cost of switching.
While we're at it, I think that there's another misconceptions in this interview.
programmers are programmers because they like to code -- given a choice between learning someone else's code and just sitting down and writing their own, they will always do the latter
Two of the five developers at my little software company are programmers because they like to figure things out. So we almost always figure someone else's code out before we do anything ourselves. There are varying degrees of this in a lot of the developers we've got there. I would say that none of us will write anything ourselves unless it saves us a considerable period of time.
But even more, if you had a relative who was always wondering, "What is it that you do all day?" you could hand my book to that relative and say, This is what my work is really like.
No. I couldn't. My experience as a developer is nothing like what he's described. And he didn't talk about the phenomenon of unknowns that I've noticed - for every project I do, if I estimate how long the known things will take, dealing with unknowns will generally take 60% longer (so multiple time estimates by 3 is generally correct). He didn't talk at all about testing.
Almost everything he talked about are things that I thought would be true when I started but that have ended up more or less untrue. Discipline coding makes a difference. Automated unit testing catches most problems, and regression testing finds almost all the rest, and not everybody does these things.
Parent
Nine women cannot have one baby in one month (Score:5, Funny)
It needs more professionalism (Score:5, Insightful)
Whatever testing is done often only tests that the product produces the correct answers when feed the proper input - no account is taken for how the program reacts to incorrect or incomplete data.
Changes are requested faster than they can be implemented and often are not communicated very well.
In short there are systemic failures throughout the whole process, from inception through to delivery. There is no single answer to why software is hard and there won't be until the industry matures and people start to get thrown out of the business for acting unprofessionally
Re:It needs more professionalism (Score:5, Informative)
I get requests like we need a program that users can run from the web that shows them if batch scans are in balance. I am then left to resolve on my own things like:
*What is a batch scan
*Where can one find a batch scan
*What does it mean for it to balance
*Can the thing just print a big Y or N in size 48font on the screen or does it need to detail something
--Then comes the anticipatory stuff
*Is the user going to expect to be able to correct an in blance
*Now that I understand what a batch scan is *maybe* I see all this other stuff should I also report on those things
*etc
People like to expect programming to go like engineering or architecture. Its not the coders don't want to apply discipline to their craft its that they can't. Nobody would dream of approaching an engineer or an architect without a pretty good idea of what they want to do. That is not to say the professional is not going to have to help them work out most of the details but the basics are going to be pretty clear. Imagine if I sent a request to an Architect asking for a "Structure that will be used by people." and gave no other information. I really doubt I would get a call back.
You can argue the PAs are not as good as they should be about requirements gathering, but I think there could be much more professionalism on the part of requesters as well. Its a waste of my time and theirs when they engage me as a developer before they have put even the most basic thought into what it is that they want. Some of this stuff is really esoteric and I understand that I will have to help them figure out how to solve the problem. I do feel they should know what the problem is.
Parent
Re:It needs more professionalism (Score:4, Insightful)
Parent
Programming without cookies (Score:5, Interesting)
Becuase People don't know what they want! (Score:5, Funny)
I would say the reason a lot of projects, even small ones take so much time is that requirements cannot be defined.
Compare building a house to software. Before you build a house
Schedule times can slip but you still know where you are in terms of progression.
If we built this house the way we do software development
Re:Becuase People don't know what they want! (Score:5, Interesting)
Your step one in "building a house" can go through all 6 of the steps that you have listed for software development. We get hired by clients, sometimes they have a good idea what they want, sometimes they don't. Sometimes what they want is feasible, sometimes it isn't. It's not unusual for even smaller projects to drag on for years, because the client keeps changing his/her mind. Many projects that cross our desks will never be built.
Many projects are not the traditional design phase ->building phase. They often overlap, and it's pretty messy.
I could go on for paragraphs with the similarities that I see between software design and architecture, but I'll save that for another post.
Parent
Re:Becuase People don't know what they want! (Score:4, Insightful)
Parent
Re:Becuase People don't know what they want! (Score:4, Funny)
Now imagine if every single weld was a unique, custom job that had never been done before, and if any of them are imprefect, the car crashes.
Parent
Re:Becuase People don't know what they want! (Score:4, Funny)
Parent
Software is hard AND there's lots of incompetence (Score:5, Insightful)
Incompetent developers tend to make things more complex than necessary. From that point on, under economic pressure, workarounds are needed to get things done. This in turn makes things even more complex than necessary. THAT is what makes writing software hard. The problem is, it is difficult to be aware of the skills that we lack. As such, a lot of programmers with a huge ego don't deserve one.
I'm not into Extreme Programming per se, but I've noticed that if multiple people look at a piece of software, chances of problems going undetected get smaller and smaller. Yes, even if you, a master programmer, show your code to a rookie, the chance of bugs going undetected will reduce. In fact, it will inevitably result in more bugs being detected before rolling them out to customers.
First Post! (Score:5, Funny)
Product managers... (Score:5, Informative)
Product managers I have seen (and I have seen many) often don't know zilch about technology, but even worse they usually also don't know much about their market, target audience/users, User Interfaces, project management, etc.
Consequently they simply don't know what they want and aren't able to explain it in one coherent paragraph of sentences. Once they would be able to explain it, the actual coding would be half as bad.
So if this guy complains that their projects back in the days at salon went bad, I'm not suprised. He's not a coder after all, he was a typical clueless product manager - started out as a journalist and suddenly he was responsible for a type of product he knew nothing about: CMSs, in addition to having no other qualification in software development or a related area (UI design, project management).
So am I surprised this project didn't succeed? LOL, of course not.
You wouldn't let a journalist build a space shuttle or a car now would you? But software? Sure, software is easy, anyone can do it. In the end, it's probably not harder than building a car, but not easier either. it just takes proper skills for all roles in the team, is all.
Too many ad-hoc hacks (Score:4, Interesting)
I think that once someone improves the situation of software architecture and programming languages so that programmers don't have to mess with ad-hoc hacks but instead write the logic that they want to implement, then software will cease to suck.
The main problem is Operating Systems architecture and Programming Languages.
Due to lack of time, I will only list a few of the Operating Systems problems that weren't solved after more than 30 years of OS development:
This would also get rid of the unnecessary bootup/shutdown sequence all programs are currently dealing with.
Instead, a capability-security model should be used (not POSIX capabilities, but EROS/KeyKos type ones), which is much simpler to use, verify and much more powerful and fine-grained. This would also facilitate secure movement of components between computers - which could be done automatically by the OS to improve performance. More on that on a later post.
Re:Too many ad-hoc hacks (Score:4, Interesting)
1) How do you know a GUI application from a non-GUI one? What about programs that are run locally, but viewed remotely, and vice versa? What constitutes a "GUI" application?
2) But you are allocating different types of "memory"! See Leaky Abstractions [joelonsoftware.com] for more information on this. Your "everything is memory" model sounds nice, but lacks a few key components.... When I fclose() a file, I have a STRONG assurance that the file has been saved and wouldn't go away if the power failed. That's not the case in your "everything is memory" model...
3) You are either talking about a security nightmare or pixie dust. How does computer B know that it's OK to run code from computer A? See other comments on #4
4) Capability security requires somebody to set up all those !#@!@# permissions. POSIX, by contrast, is very simple and requires little effort to maintain. Is POSIX ideal in all situations? No. But it's adequate in most circumstances without a lot of effort, and it's usually better to have a "just barely suits" possibility with a decent default than a perfect possibility with a lousy default. Perhaps that explains why your touted EROS operating system died on the vine?
Parent
Good programming is a boundaries problem (Score:5, Interesting)
Unlike most engineering projects that are completed and done, most programming is a living growing process that is constantly changed modified and improved.
That implies that there is a need for specialisation and clear boundries, to assign "ownership" or "territory" over certain parts of code. A programmer who understands it and gets the last say on how it's changed and have clear non-arbitrary rules for changing that "territory". Like in open source projects. If you want a kernel fix, you submit it to the proper maintainers, or make your own fork, but no corporate bureaucrat comes along and micromanage how the code is merged and managed.
Re: (Score:3, Insightful)
Most engineering projects (and software projects...hell, any project) are like children. They're never completed and done. Once you give birth to one, you're stuck with it for a long time....
Software is neither "hard" or "easy" (Score:5, Insightful)
I have a large, global project underway. User requirements are done and have been done, and we're turning those requirements into things we can code or deliver ("View a workorder", "Print asset detail", "Group revisions into single document"). Of that, we have 150 odd deliverable items, not to mention all the fit/ finish work we may have to do, and all of this barely touches on reports, security roles for users, etc.
The reason we're going to make our date, despite the 1280 discrete requirements we need to test, is that we've taken the time to look at the requirements from a few different angles and come up with a solid design plan, before even thinking about implementation. Each piece will build on another, really hard parts are identified early, blockers and such are flagged ASAP. We know things will emerge that we didn't expect, but we've got the biggest chunks identified and working together on paper. We have the flows mapped out, exceptions and variations listed, and a user group that has to sign off on every iteration of the incremental build (we're spiraling out functions and features).
The only thing "hard" about all of this is the incessant thinking about the details, and discipline required to focus on the un-fun part of software construction, i.e. the planning and design walkthroughs. The itch to code something already is growing, but delayed gratification means that when the time comes to actually write something, the design will almost certainly lead to a working, if not optimal, solution. We can refactor as we go, but it needs to work completely before it can work efficiently.
I've been following Chandler off and on, somewhat through Spolsky's references to it and some stray links around the web, and sounds like design didn't go deep enough into what it'll really take to build some of the pieces.
-BA
Once again... (Score:5, Insightful)
Re:Once again... (Score:4, Interesting)
This thread is full of analogy after analogy.
I think if you want to have a discussion amongst people who develop a software you have to ditch the analogies, because none apply. The reasons software development takes longer that you might think, or the reasons software is difficult to create, sometimes doesn't give the expected return, sometimes is buggy, etc, doesn't have anything to do with cars or buildings or spacecraft.
Parent
No Silver Bullet (Score:5, Informative)
No Silver Bullet: Essence and Accidents of Software Engineering [wikipedia.org]
Because people forget experience (Score:4, Informative)
This author had to discover Fred Brooks after he'd started a career of big projects. TMM should have been in his school curriculum.
Mythical Man Month (Score:5, Insightful)
Why must we rehash this over and over again.
If it were easy, then yes, we would have push-button frameworks that magically created programs. What bothers me more than the ignorance of the "no silver bullet" mindset, are the pushers of programming environments that supposedly will solve all our problems. Bullshit. Corba sucks. EJBs suck. All these things suck. Just write good software and stop looking for the golden chalice.
It's about people (Score:4, Informative)
By the way, I read the article and this paragraph:
Disclaimer: Scott Rosenberg was responsible for Salon's hiring me 10 years ago, was my editor and boss for many years, and is a close personal friend. My daughter baby-sat his twin sons as I "interviewed" him. So I'm utterly biased and completely partial. But "Dreaming in Code" is still a darn good book. - is one of the reasons why software sucks. I have seen too many cases of friends hiring friends not because the people being hired are the best that could be found (or that interviewed) for the job, but because they are friends of the friends. This touches on all avenues of life and business, the best people for the job are not hired, because friends of the friends are hired and those people more often than not are not going to do the job right.
One should either recommend a person for a position or interview the person, but not both.
darn customer wants and needs (Score:5, Informative)
A person needs to cross a river, and so believes a bridge needs to be built.
What he should be doing is ask: "I need to cross this river, can you help me?"
Instead, he asks someone to build a bridge, a bridge of this and this dimension so his particular vehicle can pass, and a bridge with this and this feature because "it would be nice if..". He also wants a bridge that looks in this and that way, because since he is paying for the bridge he wants it to look the way he wants it to.
First he was crossing a river, he is now building a bridge. Will the bridge help him cross the river? Who knows? Maybe a bridge like he imagines can't be built, or only be built poorly.
Its the same thing with software. People want the strangest things. In fact, what they WANT is the only thing they know. They don't know what they NEED.
So you get an organisation, a business, with an office that has a problem, any problem. The problem will ALWAYS occur because some OTHER process isn't working somewhere else in the organisation. You get bad data from here, and the clerk is expected to output good data out there. In the 50's I bet they wanted more filing capability or better typewriters to solve these problems, in this day and age they want IT. They want a system!
So instead of using excel, notepad or even a piece of paper like any other sane human being would to keep track of the information, a yell echoes down the corridors: "We need software to support our business."
So some programmers are hired. They get a description of the problem, which isn't logical in the first place, and they are expected to solve it. Their tool, software, is built using a logical language. It is used to describe the data, and solve the problem by adding a few flows for that data during certain conditions.
So the programmer (or bridge builder) sits himself down. The first 10% of code/thought he outputs is usually all that should be done about the percieved problem. That is, a description of the Need.
The next 90% of coding is about the programmer trying to coerce a logical language around non-logical flows and non-optimal solutions, hammering a square button into a round whole, with GUI's, buttons and special extra functions for special extra cases, and those extra wants on top that really describe other problems.
So we will end up with a mishmash of buggy code that describes the wants of the customer. The Want to solve a problem that should have been solved with organizational changes, or changes in the work processes. But hey now everyone is happy again, software is supposed to be a bit buggy, the organisation is obviously still working non-optimally (software can't fix that), but at least the clerks now have webpages to input the bad data in as long as the servers are up.
Optimally, the programmer (yes the programmer) should look at the problem, trace it down the whole organisation, yea trace the customer Want to the REAL problem and the REAL need, proceed to make the organizational changes and be done without an IT system at all.
We all know that won't happen, but that is what should be done. Don't expect a logical function (software) that describes a non-optimal situation, to function in an optimal way.
Software doesn't work that way, thats why its hard.
Re: (Score:3, Insightful)
Re: (Score:3, Interesting)
"we need to listen to what Brooks said... more specifically their knowledge and experience."
Basically what I said.
"Solid developers will produce solid software."
Ibid.
Re:Many "software engineering" books are trash. (Score:4, Interesting)
I sometimes wonder if it's a questions of people who naturally see things as images, versus people who don't.
Anyway, I'm with the grandparent. A small class or state diagram can be useful to me, but I get lost very quickly in a big or detailed one. And when I go into details, I soon find that what I want to say is easier to express in text, where I am not limited to the few languages permitted by the UML.
Parent
Re:Why POOR coding is REWARDED!!!??? (Score:4, Insightful)
Sorry to break this to you, but apps today are not faster than the ones they replaced. We used to write more efficient code, and had more spartan interfaces, because we had to. But I don't think Microsoft Office today is noticably faster than the WordPerfect / Lotus 1-2-3 type apps we wrote in assembler under DOS. All that extra CPU power is more than eaten up by layer upon layer of slower and slower software.
All this allegedly to make programmers more productive. I haven't seen that either.
rd
Parent