Has Software Development Improved? 848
earnest_deyoung asks: "Twenty-five years ago Frederick Brooks laid out a vision of the future of software engineering in "No Silver Bullet." At the time he thought improvements in the process of software creation were most likely to come from object-oriented programming, of-the-shelf components, rapid prototyping, and cultivation of truly great designers. I've found postings on /. where people tout all sorts of design tools, from languages like Java, Perl, Python, Ruby, and Smalltalk to design aids and processes like UML and eXtreme Programming. I'm in a Computer Science degree program, and I keep wondering what "improvements" over the last quarter century have actually brought progress to the key issue: more quickly and more inexpensively developing software that's more reliable?"
Open source and collaboration (Score:3, Interesting)
Functional languages (Score:5, Interesting)
I haven't programmed much lately... (Score:2, Interesting)
I can say I honestly don't like Java.
Nowadays we've got great tools like Flash, scripting languages like VB Script, and markup languages like HTML.
Not every programmer these days is a old COBOL nerd, ASM coder, or C junkie.
I yearn for the days when Borland was great. The Turbo C++ and Turbo Pascal products probably got half of the programmers in the 80's, late 90's started.
Progress Balanced By More Demands on Developers (Score:5, Interesting)
However, I think they're equally balanced out by huger demands on programmers. Once it's realized that a programmer can do 2, 3, or 10 times as much work by using more efficient methods management is quick to pile on 2, 3, or 10 times as much work!
This isn't really unique to programming either. I think it's universally applicable to any area where technology permits greater productivity.
For example, look at all those ads from the 50's. Things like the microwave, the vaccuum, and the dishwasher were supposed to usher in a new era of leisure. Do we have more leisure? No, we have less, as those luxuries become necessities and we cram in more activities in out new-found time in order to stay competitive.
The Software Process (Score:2, Interesting)
"I'm in a Computer Science degree program"??????? (Score:2, Interesting)
I'm in a Computer Science degree program
If you're in the U.S., GET OUT WHILE YOU STILL CAN - the tech job market (and the economy in general) are in horrible shape. My friends are coming out of college with CS/IS/MIS degrees and finding NOTHING!
And if you stay in CS, may God have mercy on your soul.
eXtreme Programming == NO (Score:4, Interesting)
1) Write your test cases up front... this is ages old. XP isn't as rigourous as others who say "and make sure other people write them".
2) Pair Programming, works for two people of equal ability. The two headed surgical team from the Mythical man month is a much more effective way of using two heads.
Basically things like XP sum up how long computing has to go to become an engineering discipline. In every other engineering subject there are critical elements:
Requirements
Design
Testing and approval of design
implementation
testing of implementation (throughout implementation)
Delivery.
Maintainance
For a construction project all of these elements are mapped out well in advance, which is why the construction industry can work on lower margins.
To become better requires not a "Silver Bullet" as Brookes says, the technology won't make the improvement. Its actually about people applying the rules _rather_ than looking for the Silver Bullet. Some projects succeed, others fail, there are reasons for the failures and the successes. But rarely do we learn from either.
XP is the embodyment of the non-engineering approach to computing that pervades this marketplace. The idea that you can build it wrong and change, don't design "code and check", have a unit test written by a bad coder to check his own bad code.
Brookes is right. At the end of the day computing success is down to a realisation of the soft-skills allied to technical talent.
If you have 10 brilliant people leading 100 average people... fire the 100 and support the 10 to do the delivery effectively. Make sure they follow a process, and make sure that the requirements are defined and change as little as possible. Make sure designs are verified, make sure code is reviewed.
Sure its less exciting that "just coding" but in the end it takes less time, costs less to maintain and delivers what the customer wants.
Engineering is a discipline, XP is just glorified hacking, only by becomming disciplined will software improve.
Software improvement in the last 25 years? (Score:4, Interesting)
Of course it is now easier to create software than before.
First of all, source management software wasnt available 25 years ago. Try creating a huge piece of software without any way to rollback changes, share the same source tree with other developpers, etc... (cvs/sourcesafe/starteam/etc)
Second, profiling tools. Hey, you want to know where that memory leak is? Where that CPU bottleneck is? Pretty hard to do when you were coding in cobol many years ago... Doing the same is way easier now with OptimizeIt and stuff like that.
I could go on and on but I must leave for work =)
One day... (Score:5, Interesting)
I've always believed that one day some bright college kid is going to come up with a completely different style of computer language and interface, and when that happens we will all slap our heads and go "D'oh! So that's how we should be doing it! Obvious!"
Like the web and P2P, the most influential ideas are often quite simple, and "obvious".
Here's a thought... (Score:3, Interesting)
perl is ugly to code in, and perl OOP is obviously a hack. I had a graphics/OOP professor who said, "nobody likes to program in perl, but it gets the job done." Obviously he lives in the land of language theory, where perl doesn't, but it gets the idea across...
perl gets the job done because of its massive collection of components.
I think I'd go further to say that the big improvement there is in repositories where you can get massive collections of components, as there have been languages like perl (in terms of having lots of stuff - PL/1 comes to mind) in the past.
Places like CPAN, Sourceforge and Freshmeat really make the difference. So ultimately, the internet is the means through which software development has sped up (at least when you're not talking about RAD-GUI development, which is another thing entirely).
Re:CVS / RCS - the next step? (Score:2, Interesting)
I was recently discussing this sort of thing with some friends and got into what I would love to start as a project - something to the effect of "fvs" or function versioning system - which would allow me to keep tabs on "just-a-box" functions which i use throughout my programs.
I think any programmer who sees the benifts of CVS would understand where im going with this concept. We all have functions we use again and again - and realizing that there is a potential flaw in a given function at one point is always followed by exasperation because one realizes that the function needs to be changes in X number of programs.
Object Technology.. (Score:3, Interesting)
Design patterns also play an important role in allowing for a given design to be re-used, consumed, whatever the case may be. OOP related technologies such as UML, Corba, and now many XML based solutions are beginning to mature the field.
I am not as experienced as I would like to be with OOP, but I can say that I have been in the procedural world long enough to realize that there seems to be a divine power in OOP. It makes you *think* entirely different about problems, and by breaking a very large, very complex problem into approachable components. Allowing not only a single developer to build more complex systems, but systems that are well suited for re-use in future or existing systems.
Observations (Score:4, Interesting)
What I have observed in the course of software development, in various companies, is that the management dictates that "thou shalt follow said process" and there will be the obligatory design reviews, spiffy diagrams, an all the huffing and puffing. But when the smoke clears, it still comes down to 9 months of work compressed into 3 months of actual work time, and everyone shifts into a hack 'n slash mode. The processes in place fail because of a lack or adequate time and inflexibility of deadlines.
Away from algorithms? (Score:3, Interesting)
Patterns... (Score:2, Interesting)
Re:Progress Balanced By More Demands on Developers (Score:5, Interesting)
Today we have JDEdwards OneWorld. Written in C++ and other state-of-the-art tools. The distribution runs about 10 GB, with a working set for development purposes of 1.5 GB. Its function? To provide accounting and manufacturing management support for manufacturing companies. How well does it do? Is it any better than ManMan? I will leave that for you to decide. But hey - you can cut-and-paste directly from OneWorld to Excel. That's a gain I guess.
sPh
It is all about passion.... (Score:2, Interesting)
Software (design and) development is all about passion. Only mechanisms to express the road to success have changed (and improved). Experts look for species like good software developers, analyse them and define a new process. And most of them are not wrong.
There are hardcore believers of waterfall model. And so goes with extreme programming. Configuration management is luxary (and waste of time, sometimes) for somebody, and may not be same for others. The best part is, we have methods suitable for individual's nature.
Progress in software engineering (Score:5, Interesting)
Brooks, in the "No Silver Bullet" essay referenced above, stated that there is both essential and accidential complexity in software development, and because of that there never would be a "silver bullet" to slay the software "monster". However, there are fundamental practices that increase the likelihood of success and fundamental pitfalls that every project faces. And, in the end, the root causes of most failed IT projects are human factors; in fact, you could just cite the "seven deadly sins"--pride, envy, gluttony, lust, anger, greed, sloth--and probably hit the nail on the head.
In conjuction with that, far, far too many practitioners in the IT field lack one or more of the following:
To quote George Santayana (who is often misquoted):
Software engineering is hard enough--with all the human issues--without further handicapping ourselves with ignorance of all that has been already discovered and documented. Yet that is exactly what most IT workers do. Until we find a way to solve _that_ problem, the failure rate for IT projects will remain high indeed. ..bruce..
Opinion: Yes. (Score:3, Interesting)
I think the ingredients you mention have made production of software a better process over the past 25 years. Software applications can do more and can be built more quickly as result of those improved tools available to developers.
However, you still see a great deal of unreliability, bloated, and inefficient code because developers are trying to do much more than they did 25 years ago.
If all we needed to do was re-create the applications of 25 years ago, then the benefits of new techniques would be more evident. But people demand more and programmers want to create works up to their full personal potential and exceed what is currently possible.
Ragged-edge software is manifest evidence that we still are constantly crossing the barrier of human potential, that place where what is barely possible becomes what doesn't work. It's a good sign of innovation. And, it provides added impetus to keep trying to find more ways of improving the software development process.
Re:Is it engineering, or not? (Score:2, Interesting)
Yes. Good software does. In my opinion, this is the most significant problem with software today. There are too many "software engineers" that don't follow any sort of engineering principles at all. Quality and documentation are afterthoughts. Design consists of a conversation or two before implementation (coding).
If this sounds like you, please leave the industry now. We've suffered enough.
It isn't engineering yet, and won't be for a while (Score:5, Interesting)
If requirements were as poorly understood and changed as fast in the physical world as they do in the software world construction would cost a fortune and most big buildings would never get finished (or would never be fit for purpose).
People who say things like "Make sure the requirements don't change" are living in a fantasy world where they want to blame their inability to deliver on someone else.
The rules haven't changed, get a high quality small team, get good access to a user who knows what they want and grow a system from small beginnings, checking at each stage that it all works and that quality is high.
Its all there in Brooks.
Its no surprise that the guys pushing the agile methodologies were all very succesful designers and developers anyway.
Re:Absolutely! (Score:3, Interesting)
There are many. Others have mentioned Komodo [activestate.com] and the Visual Perl plugin [activestate.com] for Visual Studio.NET.
There's also Perl Builder [solutionsoft.com], which people rave about but I have not tried. They claim to be the most popular Perl IDE.
I would also like to add that an Open Source one, Open Perl IDE [sourceforge.net] is decent. I use it at home.
I use Komodo at work (because it can act as an IDE for other languages like XSLT) and really like the Perl and Regular Expression debuggers.
-bp
Re:Cheap, Good, Fast - Take all 3, if you're good (Score:4, Interesting)
On the other hand, it's _probably_ as cheap as good and fast is gonna get.
Re:Cheap, Good, Fast - Take all 3, if you're good (Score:2, Interesting)
Re:Cheap, Good, Fast - Take all 3, if you're good (Score:1, Interesting)
is not methodology (though it helps) nor
good process (though that helps too.) The key
is great people (or taking good people and
making them great.) In software engineering
the difference in productivity between an
average programmer and a good one can be as
much as 1000%, and between a good and a great
programmer another 1000%. That means that a
great programmer can do more in a week than
an average programmer in a year.
That also means that if you have to pay the
great programmer $1000 an hour, then it is
a bargin. However, great programmers are much
cheaper than that. Usually, pay them third
quartile, a fat bonus, as much computer
equipment as they want, and give them interesting
challenging work in a nice work environment,
and they will do what comes naturally to them.
One other thing, it is better, IMHO, to get
fast programmers than good (quality) programmers.
People who produce very high quality code slowly
are quite hard to train to become quick at doing
so. Fast programmers can generally speaking be
easily taught how to produce good quality code
without impacting on their productivity. (Great
programmers usually hack macros to do the boring
stuff for them.)
For our student freind, I would tell him/her that
if he wants to become a great programmer listen
to all they are teaching you, but far more
important is to code code code. Write lots of
programs, way more than the dumb ass homework
assignments they give you. Also, read as much
great source code as you can get your hands on.
There is lots of it available under GPL.
Learn from the masters, and practice is the key
to a skill like computer programming. It won't
matter at all if you forget the seven layers of
the ISO communications model, but if you don't
know the key idioms to writing great code to
the point that they are muscle memory, you'll
never be a great coder.
not a money pit... (Score:4, Interesting)
Johnson and Rich were also proponents of having Lockheed engineers maintain their aircraft instead of having military personnel do it. Using Lockheed engineers would save a ton of money because they aren't rotated and therefore don't need to be retrained. Sure, Lockheed would get some money out of the deal, but not as much as the gov't would save.
The Skunk Works wasn't perfect, but overall they were pretty good with money under Kelly and Ben.
Re:Cheap, Good, Fast - Take all 3, if you're good (Score:2, Interesting)
They don't have to be the best developers on the planet IMHO, just competent. I work for a company where the leadership has gone from a vapourware touting salesman, to a still slightly too ready to believe that the developers are all wrong, because now the major powerhouse is the support/customer service organization.
What really would have helped was STRONG capable development management. None of the people I work with are stupid, they've just been made to believe they are for so long, some of them have bought into it.
They badly want to rearchitect core products, but are continually dragged in several directions through mismangement.
Anyone is capable of brilliance, but they need an opportunity to fend off the naysayers while they accomplish it
what about AOP? (Score:1, Interesting)
aosd.net [aosd.net]
Re:Programmers *are* the problem (Score:3, Interesting)
Re:Best improvements (Score:4, Interesting)
Actually a lot of people would. The quality of code generated by the SUNpro and MIPSpro compilers on SPARC and MIPS processors respectively leaves GCC in the dust. GCC really only comes into its own on x86, because Linux (or *BSD) on x86 is the platform that it gets used most on. GCC is portable, yes, but it isn't built for compiling high-performance code. So you need to ask yourself whether getting binaries that execute 2x as fast is worth using a slightly less well known compiler for.
Literate Programming (Score:3, Interesting)
DEK has since written an entire book on the concept (_Literate Programming_ a CLSI series book) a decade ago, but one seldom sees source so provided.
There are some really cool example programs which're quite interesting (and educational) to read, for example:
Will Crowther's game Adventure - available here: http://sunburn.stanford.edu/~knuth/programs/adven
Or a CWEB version of the RPN calculator for K&R's C Book: http://www.literateprogramming.com/krcwsamp.pdf
Probably what really needs to happen is a way to post a program as a web page, then to click on a link on it, to automagically compile and run it....
William
Re:Away from algorithms? (Score:4, Interesting)
From one point of view, only a tiny minority of today's programmers ever need to create "new algorithms"- so much documented study has gone on in that area, you'd only be reinventing the same wheels.
That can be a sign of maturity, that the field has evolved to the point where specialists can go their thing, and not force everyone else to understand the trickier aspects.
Looking at your examples- data converters particualrly- I don't see much room for new algorithms. From a restrictive, Computer Science standpoint, nearly everything you do will be isomorphic with a known existing algorithm (modulo "trivial", "cosmetic" variations, of course)
Naturally, people who don't understand algorithms will have difficulty selecting the ones they should re-use, and are at risk of using them wrongly. But that's a consequence of newer, user-friendly development tools- with a lower barrier to entry, less competent persons can enter the profession, and still muddle by.
Re:Java and GPL - slower software, less incentives (Score:3, Interesting)
Most companies see the small performance loss from using object oriented languages does not compare to the loss in man hours maintaining non-OOP code. OOP code scales better when that small application gets larger. Code reuse is simpler and can aid in quicker development of other applications with similiar functionality in the future.
The "vanilla unix" does not exist anymore so its not really an issue. The original idea of byte-code compatiblity was lost when unix commercialized. This spurred the development of a language like Java which allows code to be cross-platform. Writing in ANSI C would be another way of doing this if only it was implemented the same across the Unixes and the hardware remained a constant. That will never happen.
Add to this GPL/OS that slashes meaningful business value from well engineered software components
I think that there are many well engineered software components under the GPL and could give plenty of examples. These components required the experience and time of many programmers. By sharing this codebase IT departments do not have to hire dozens of programmers to create a similiar product.
You may be a "professional" developer, but I think you should avoid any position where you would have decisions on the direction of what technology is used within your company until you come into the realization of the economic savings of GPL code and the reasoning behind object oriented languages. These subjects are generally covered in most computer science programs at most universitys.
There are some really amazing things going on... (Score:2, Interesting)
People are still the limiting factor... (Score:3, Interesting)
Of people and projects and the use of tools and technologies:
Let's see... Two contracts ago I was on a team of developers who had one hammer in their toolbox: COM. I pleaded with them to limit their use of COM but they went on a rampage. They wrapped ADO in a COM based object/relational DB abstraction. Then they used the COM based DB wrapper in their COM based business tier to talk to COM based Active-X controls in the UI. I pleaded with them to use MTS, distributed transactions, and stored procedures as the system was a distributed client/server app; but I was told that transactions and stored procedures would add too much overhead. COM, COM everywhere and no transactions in sight! Result: A slow, giant, buggy, leaky, unstable 4 tier COM based pig of a system that has major database issues. I finally couldn't take it anymore and so I gave notice after 1.5 years on the project. I left there 2 years ago and today they have sold the thing to one customer - who is extremely unhappy...
My last contract was an aircraft carrier combat system. 220 developers had been playing in Rational Rose for 1.5+ years when I showed up. They had modelled over 5,000 classes and had not modelled anything called "Weapon", "Sensor" or "theShip". They used Rose to generate their C++ code for HPUX and VxWorks and they used TCL/TK to do their UI. Frankly, if there a was a fork in the road these guys took the hard way every time. Result: the system is an impossibly huge nightmare that runs on dying platforms and depends on a dead UI language. The company has run out of money to finish the system. The staff has been cut from 220 to 50 and no deadlines have been moved. None of my friends who remain on this project work 40 hour weeks...
Currently I'm on contract to do a UI by X-Mas. I just fought a 2 week pitched battle with pig headed engineers over whether or not we should use C#, VB, or MFC to do the UI. I finally ended the battle last week when I walked into a meeting with working screens coded in C#. The UI is mostly finished and all are very impressed with my work. But I have been unable to convince unwilling, MFC loving, pre
I guess what I'm saying is that the biggest thing I've run into in the last few years is people who have very real design decision making power but who don't have a good background in what tools and technologies are available. I see projects blunder into Death March situations because the people working on them are unwilling to keep up with technology or are unaware of what tools are out there...
I work like heck to keep up with the times. I wish more people would read books like "The Pragmatic Programmer" and take them to heart...
--Richard
We keep 'improving' irrelevant things (Score:2, Interesting)
churned through imperative programming languages that do the same thing with just a bit different syntax and Spartan/or complete set of standard libraries (Cobol, fortran, C++, pascal, java, C++, C, perl, C#). So we have wasted investors money,
customers money, many man hours by lying to them that when we use the new programming languages things going to be much better (that self-satisfying process is now called 'commercial software industry').
In addition to manual conversion to different lang. syntax (and fixing some bugs on the way) of the existing software, we spent millions of dollars on what we call 'Design Tools' and those are mere lego-like packages that allow us to break-down our tasks into such small and disjoint components that a developer who was not part of the 'team-that-did-the-breakdown' would need to spend weeks to find out what set of 'components' (an important word in our vocabulary) enables that button (and it turns out that the a row in a database is read by the 'database layer', then the there is code that the checks if that row is has all the necessary data, then it sets some boolean flag (we call that a business logic layer), then the button checks that flag and to be enabled or disabled (and that we called our GUI layer)).
In the last 10-15 years, we came up with a role of a 'Software Architect' -- that probably had a noble goal, but really degenerated into a task of dowloading the latest 'new' technology from the internet, then setting them up for some irrelevant test, and then running around the company and telling everyone how we should 'upgrade to the latest technology for some great benefit and no one can understand'.
Then, in the past 20 years we convinced our customers that they cannot really expect bug-free software from us, but instead we can do something that does most of what they want and fix the rest later. So then the customers simply stopped believing us that we can manufacture something that works, the only thing that customers can do is to request us to do something by some date (therefore the unreasonable (but always movable
Look at the press releases of the public software companies: they do not mention what functionality their products have, how fault-tolerant they are... instead they mention what Tools they used to build their stuff (XML, EJB, Java, OO, UML, web-enabled and wireless ready). And for a while the stock market rewarded just that. But have you seen a profitable hardware design and manufacturing company bragging in their press releases what compiler or VLSI design tool they use, instead of highlighting the functionality of their product?
-----------
We also convinced ourselves that there are no 'methods' to desing software without bugs, instead we'd rather rely on QA to test it -- because we think software is like nothing else -- it cannot be VERIFIED for CORRECTNESS by us.
And that is fundamentally what needs to be changed: we must be able to use mathematically sound algorithm verification techniques and we should ask our tool vendors NOT to deliver to us a syntax change every 3 years, but build for us tools that would do what do data-flow analysis on the program logic level, that would simulate timings and parallel execution for concurrency enabled applications (threading/etc) and so on.
-----
Of course there has been a lot of progress in the computer science and mathematics that supports it, but most of it did not make in to commercial app development world.
Compilers are better, language parsers are better, OSses are better -- but those are the tools where the science was used to design algorithms (like graph coloring for register allocations, or other optimization techniques)
And hardware, of course is much better --
the commercial software 'looks' like made leaps and bounds -- but it is really some of tools that we use underneath that made us look better.