Bad Software Runs the World 349
whitroth tips a story at The Atlantic by James Kwak, who bemoans the poor quality of software underpinning so many important industries. He points out that while user-facing software is often well-polished, the code running supply chains, production lines, and financial markets is rarely so refined. From the article:
"The underlying problem here is that most software is not very good. Writing good software is hard. There are thousands of opportunities to make mistakes. More importantly, it's difficult if not impossible to anticipate all the situations that a software program will be faced with, especially when — as was the case for both UBS and Knight — it is interacting with other software programs that are not under your control. It's difficult to test software properly if you don't know all the use cases that it's going to have to support. There are solutions to these problems, but they are neither easy nor cheap. You need to start with very good, very motivated developers. You need to have development processes that are oriented toward quality, not some arbitrary measure of output."
Nothing New (Score:5, Insightful)
That is because corporate infrastructure software does not generate revenue. Why spend money that does not directly impact the bottom line?
Maybe when you get people who actually understand the underlying business rather than a MBA graduate, that will change.
The old adage (Score:5, Insightful)
Re:Numbers don't lie (Score:5, Insightful)
It's a big world (Score:5, Insightful)
There aren't enough 'good' coders in the world to implement all the software that needs to be written, let along 'very good' ones. Not to mention good architects, designers, requirements analysts, etc, etc, etc. And even if there were, software that needs to work together isn't always designed to do so. Hacks, cludges, and jerry rigged solutions are what hold the tech world together, no amount of wishful thinking is going to change that.
Re:The old adage (Score:5, Insightful)
Another rule here is out-of-sight-out-of-mind: If management can't actually see the effects of what's going on, they don't care how good it is, which is why UIs can be fantastic while the backend completely sucks.
Fixed (Score:5, Insightful)
That is because corporate infrastructure software does not obviously generate revenue, and losses of opportunity are invisible.
Fixed it for you.
Basically just supporting the last half of what you said.
Re:Cost of geek food going up (Score:1, Insightful)
Re:Numbers don't lie (Score:5, Insightful)
That depends on your definition of average, mathematically speaking that's not true. What percent of numbers are below average in this set: {1, 1, 1, 1, 1000}
This isn't pedantry, this is a meaningful distinction: I expect the amount of good software is extremely outnumbered by the bad, and even good software developers can be forced into kludges by time pressures, bad team culture, etc. I don't see any reason to think that code quality globally resembles a normal curve.
Re:Yeah, but how do you measure 'Quality' (Score:4, Insightful)
Re:Numbers don't lie (Score:2, Insightful)
Re:Numbers don't lie (Score:5, Insightful)
Raw defects doesn't indicate quality. A defect by which the system occasionally has to stop and replay some data write-out because of some hoakey disk driver is not a gerat problem: the disk driver is buggy, and is using a shitty hack to fix it. By contrast, a much better written driver with a very corner case race condition that 1/100 as often simply destroys a ton of data has a huge problem.
Linux is like that. If a hard disk drive starts to not respond, it'll send it a reset command and continue. It'll mount the filesystem read-only without special options; in some conditions that's important, because the OS view of the FS might be completely different due to undetected write failures. In any case, it's still up and you can get information out of the kernel. I've had the system hose itself so bad I couldn't actually read the logs or run dmesg, but if your boot process copies a few utilities into a ramdisk and sets tty1-5=login tty6='chroot /recovery login' you should be able to switch to that tty and run. Bonus points for statically linking chroot on boot (i.e. the boot process copies everything in from installed fs, then uses ld to statically link chroot to all its dependencies), so in a barely-functional active ssh session you can '/recovery/bin/chroot /recovery /bin/sh'
A high-quality system that fails 1/10000 of the time and destroys everything is worse than a low-quality system that fails 1/100 of the time without cause for concern. Yet the low-quality system is clearly shitty.
Re:The old adage (Score:0, Insightful)
Except where Google is concerned, then it's Good, Fast, Cheap, No Strings Attached...pick three. Well, actually, we picked three for you.
Re:I've worked at investment banks since '93 (Score:5, Insightful)
Measuring how often something *doesn't* crash is extremely hard to show to the bean-counters. So, be ready to demo.
True story: When I started a sysadmin job, we had JBODs that routinely ran out of space, causing all sorts of downtime problems, like leaving the whole building dead in the water for days. I convinced TPTB that we needed decent storage (in this case, NetApp), but many of them choked when they saw both the purchase and maintenance costs. After we had it installed, I saw that a volume was close to hitting the wall. Before doing anything, I called in the most skeptical of the PHBs, and said, "Wanna see the NetApp pay for itself?"
I showed him the alarm that indicated a space problem, and told him we were on the verge of going down for two days. I waited for his skin to turn pale, and at the right moment, I said, "But we won't, because the NetApp can do this," and in a few keystrokes and mouse clicks, I added enough space to the volume.
I then told him that things like this happen once or twice a week, and I fix them without anyone knowing anything went wrong, but that I documented them in the trouble ticketing system I had installed, so if anyone wanted to know how many disasters were prevented by having decent storage, that's where they should look.
That didn't completely solve the problem of the thousand successes going unnoticed and the one failure never forgotten, but it came close.
Re:Numbers don't lie (Score:5, Insightful)
Is there a real definition for quality?
When it comes to software, yes. We have had it for decades: quality entails
Just to mention a few. There might not be a single universal definition of software quality, but there are common desirable attributes that have been known for decades. It's when people code without knowing these attributes (or not giving a shit about them) that we get the crappy stuff we get.
Software doesn't have to be perfect. It simply needs to be economical due to its qualities (again, qualities well known for decades.)
Re:Cost of geek food going up (Score:3, Insightful)
You fools think it's going to be a dime or a dollar? The cost will never stop going up, and the quality of care will go down.
Boy you libs are so damn predictable.
You want 100% coverage, gee Cuba has had that for a long time now.
"Ohh I'll gladly pay [and pay and pay and pay and pay]."
Keep bending over suckers, the real hard part is just around the corner.
No, it hasn't. Ever been to Cuba, where you get an aspirin for free, for everything because that's all they have?
A better comparison is the health care systems of say, Japan, Canada or Israel... or almost any other developed country for that matter.
Re:Nothing New (Score:4, Insightful)
That is because corporate infrastructure software does not generate revenue. Why spend money that does not directly impact the bottom line?
Marketing can always get fat funding to have designers polish the turds on the web site, but the backend people don't have access to that kind of money.
Maybe when you get people who actually understand the underlying business rather than a MBA graduate, that will change.
If payroll is threatened, you may get some action, but anything else usually gets a Band-Aid.
Re:hmmm (Score:2, Insightful)
An associate one told me that he could fix all of the problems in a $250 million dollar failing s/w project with one clip in a .45.
That is an awesome line.
Re:Indeed (Score:5, Insightful)
The quality of the stuff out there is just horrible.
Having worked under several hats, the latest being as a system architect, I can tell you exactly why this happens.
We start with some upper management who have this 'nifty idea' that they must have for the business. Ok, fine... now let's get the ball rolling!
First, you have the budgetary committee. Without any input what so ever from the technical groups that make up the technology and know what is or isn't possible, they work with vendors on a parachute budget for the project.
Secondly, with this locked in budget in hand, they introduce it to the system architects and project management. The project management are giving timetables saying 'we need this done by this time, no exceptions'. They then pass that timetable, as well as the budget, to the afore-mentioned system architect.
Introduce stroke-approaching WTF moment for the architect....
Third. The architect goes back to the project manager saying 'We can't build the specs for the money in the time allowed'. The manager goes 'oh, right'. They go to the budgetary committee and bring this up, and once they realize the bottom figure is wayyyy out in left field, they come back with 'that's impossible, we need this done, with the results of this, for the money you originally quoted us'. So... head back to the system architect...
Forth, the architect then, to un-bury himself from the absolute disaster sitting in his face, tell the project manager what will be required to minimally meet the ends. This generally requires a ton of over-seas consultants, paid to grind the wheel 24/7, at the lowest dollar, to get it to work on outdated hardware to meet the end core/cpu/memory requirements and still 'work'.
Fifth, the consultants are hired, you're lucky if they understand english sufficiently to understand the nuances of day to day communication. They also take shortcuts, because they either don't know the right way, don't want to spend time on the right way, or are told that doing the 'right way' is not time efficient for the cost. So now, we have crappy code being tossed in, usually undocumented.
Sixth, the dev work is slammed in, marginally tested, and quick-shotted through QA because the upper management are in a time crunch and don't have the time to deal with all that 'quality assurance nonsense'. So this work is now fast-tracked to production in a non-fully tested workflow.
Seventh, it's been live for a while, things break randomly without reason, but it's ok, a restart of the application always 'fixes' it. So what if you have to bounce the app every 2-3 weeks to free up that memory leak. It works, in the budget... well, maybe for a few hundred thousand more... but it's done, it provides the 'nifty feature' that the share-holders were promised by the upper management, and the things that don't work are being pushed on...
three guesses...
The management who pushed the idea? Nope.
The budgetary committee who gave the low-end figures out of their butt? Nope.
The project manager who gave the tight time-frame for the project without major input from the technical people? Nope.
I know... the IT professionals who are still at the company like the system architect, network team, dba team, san management team, and the security group who are left holding the bag of the big pile of steaming crap? Yup.
Soo... when things evidently break so bad to be noticed, and management are told to 'fix' it it will take more money, more time, and more hardware. Shock, awe, and bafflement is shown, bonuses/raises are crushed because the IT professionals obviously can't do their job right, and maybe a few heads roll because management have their golden parachute and are not held to blame for the project they initially started up.
That. Is why the 'stuff out there is just horrible'.
It's not any one thing, it's how business runs, because these yahoos frank
"Just-Good-Enough" software runs the world (Score:5, Insightful)
Remember: Broken gets fixed. Shoddy lasts forever.
Comment removed (Score:5, Insightful)
So why are they called Software Engineers (Score:4, Insightful)
I'm not trolling, it's a reality check.
Most big software projects I've seen fail hard, like millions and 10's of wasted dollars hard. By comparison you just dont see that very often in big electrical/mechanical/civil projects, which can be equally complex (eg refineries, cruise liners etc).
There are software developers with all sorts of fancy titles - architects, analysts, engineers - and yet they cant get the code right. Usually the root cause is inadequete requirements spec and failure to manage the customers expectations but that's no excuse, there are usually numerous poeple employed in the project process specifically to get those parts right.
Software engineering is still playing catch up, in the sense that most developers and development companies I've seen still dont follow a formal enough process for it really to be called engineering. Usually it's a bunch of computer science graduates having a wild stab at it, and the good ones are closer to artisans than engineers.
Until the entire software industry gets off it's high horse and admits this to itself - and more importantly admits this to the customers, we are going to continue to be dissapointed with the quality.