Bad Software Runs the World

whitroth tips a story at The Atlantic by James Kwak, who bemoans the poor quality of software underpinning so many important industries. He points out that while user-facing software is often well-polished, the code running supply chains, production lines, and financial markets is rarely so refined. From the article: "The underlying problem here is that most software is not very good. Writing good software is hard. There are thousands of opportunities to make mistakes. More importantly, it's difficult if not impossible to anticipate all the situations that a software program will be faced with, especially when — as was the case for both UBS and Knight — it is interacting with other software programs that are not under your control. It's difficult to test software properly if you don't know all the use cases that it's going to have to support. There are solutions to these problems, but they are neither easy nor cheap. You need to start with very good, very motivated developers. You need to have development processes that are oriented toward quality, not some arbitrary measure of output."

I've worked at investment banks since '93

[bobs_lounge]

and this article is absolutely correct. Forthe most part, we do regression testing, but a lot of code (a whole lot) is never unit tested, its not written to be used it tested, and there are configuration holes all over the place. Each time there is a Jerome Kerviel or Nick Leeson, a generation of auditors will come through and find systems faults, and put in reasonably effective controls, but that is not the same as programmatic correctness. Programmatic correctness often has to be baked into the code from the start (same with effective unit testing), and by and large, this is not an investment banks highest priority (as an earlier poster wrote, code that is not directly involved in revenue generation does not get funded).

Re:Yeah, but how do you measure 'Quality'

[NeutronCowboy]

AC hits it on the head. This is nothing but the age-old search for the perfect metric. Development processes ARE oriented towards quality - for arbitrary values of "quality". The problem is that quality software is like porn: you know it when you're looking at it, but you have no idea how it is exactly defined. Is it a lack of bugs? Sure, but that's definitely not the only aspect. Is it maintainability? Maybe - if the software needs to be around for the next 30 years. Is it readability? Dunno - machine code is pretty unreadable, yet there's quality machine code out there. Is it how long it took to develop, how flexible it is, how user friendly, how much power features are in it? Maybe, maybe, maybe.

Pick a metric - a boatload of metrics - and I will find you a large number of cases where the metric fails. Are we doomed? Kinda. Just like there's no silver bullet that solves all your development processes, there's no silver bullet when it comes to measuring the output of the process.

In the end, what people care about is "does it do what we need it to do?", and that's all that anyone is going to remember. Unless, of course, it's review time, and then the only thing that matters is "the metric".

Yep, we're doomed.

Re:hmmm

[PPH]

You mentioned utilities, so I've got to jump in with an anecdote.

Several years ago, I was brought in by our local electrical utility to develop a configuration control system for engineering documentation. Specifically, substation engineering documentation. The problem was presented to me as one of engineers getting behind schedule working on as-built drawings. And working from the wrong drawing revisions. So, before accepting the project, I asked to speak to some of the other stakeholders in the design and build process. Specifically, the construction crews. The first sign of trouble: Engineering management looked at me kind of funny. It seems that engineering and construction didn't get along too well. Nevertheless, as an outsider, I figured I had an advantage.

After speaking to a few workers, I had a better picture. It seems that the relationship between these two groups had been poisoned for many years by (as far as I could tell) one individual in engineering management. The crews considered him to be a raging a**hole and wouldn't work with him (he had retired years before, but people still carried a grudge). As a result, the crews pretty much built things the way they wanted instead of per the drawings. Specifically, they built new stations the way they built the last one. As a result, the smarter engineers as-built the older drawings. Those being the ones the crews were working to.

Management wanted me to come in and build a system to 'lock down' released drawings and restrict the work flow. Without understanding the root causes of their problem and how people were working to accommodate process and personnel problems. I walked away from that job offer quickly.

I have a few stories (oddly about defense contractors) where software was built intentionally to be lousy. If the users' management had to employ a staff of a few hundred people to repeatedly sanitize database entries (rather than built error checking into the entry system) it meant they were a third or fourth level manager (with the commensurate salary) instead of first level with a couple of DB admins under them.

Good software is easy to write. Bad management or personnel problems are difficult to fix. An associate one told me that he could fix all of the problems in a $250 million dollar failing s/w project with one clip in a .45.

Re:Cost of geek food going up

[Fallingcow]

Bingo.

Single-payer, for instance, would be a bigger boon to entrepreneurship, job mobility, and (actual) small businesses than any tax cut proposal the right (or left, for that matter) has put forth. Even better, if it's done right (so, at least as well as it is in the very worst system of that sort in the industrialized world, since they all spend less than we do) it'd cut costs for everyone, including big businesses.

Too bad that would be evil socialism, I guess.