When Bugs Aren't Allowed

Coryoth writes "When you're writing software for an air traffic control system, military avionics software, or an authentication system for the NSA, the delivered code can't afford to have bugs. Praxis High Integrity Systems, who were the feature of a recent IEEE article, write exactly that kind of software. In "Correctness by Construction: A Manifesto for High-Integrity Software" developers from Praxis discuss their development method, explaining how they manage such a low defect rate, and how they can still maintain very high developer productivity rates using a more agile development method than the rigid processes usually associated with high-integrity software development."

nearly unlimited funding

probably helps too :P

Re:nearly unlimited funding

And yet the reports I've seen on Praxis claim costs and schedules the same or less than the development of software of similar complexity...

Re:nearly unlimited funding

Yes, small teams help, but I think it really is worth taking a look at the tools and methods that Praxis uses because there are some remarkably good ideas in there. Take a look at SPARK Ada - the Wikipedia article [wikipedia.org] has some basic examples, and Praxis provides sample chapters of a book on SPARK [praxis-his.com] for download. Seriously, take a little time out of your day, sit down, and read a little about how they do what they do. There really are good ideas that go well beyond just "small teams" if you want to deliver correctly functioning code.

Jedidiah.

Re:nearly unlimited funding

You can't staff a whole large application development project with the best gurus: there aren't enough out there in the world.

And why aren't there? Geeks lament the fall of IT and Computer Science programs at institutes of higher learning, and wonder why people don't want to go into these fields. If there was demand for programmers of the caliber you mention and companies willing to pay salaries deserving of such abilities there would be more people studying towards such a position.

But if companies are going to just throw up their hands and say "we can't hire competent people, there aren't enough of them in the world, they only doom themselves to a continued shortfall in talent, and an increase in buggy software.

Nursing is a profession that is always looking for new people and folks who didn't make it through the four-year grind back when they were young are flocking to it because the jobs are waiting. If companies held their coders to a higher standard, they would trim some of the fat in their projects and have jobs open for people willing to do the work. The result would be more productive teams and better applications, a win-win situation.

Re:nearly unlimited funding

While it's hard to argue with the above logic, such changes don't happen overnight just because a demand increases. When we're talking about properly educating people (and by properly, I mean both intensive CS background in theory and practice, and then practical coding in real world environments in quantity to become a good coder as well), it will take five or more years for a demand spike to produce a supply rise.

The projects I am involved with today won't wait five years for programmers. This is business fact; if they aren't started now, and completed in an economical timeframe, then companies will go out of business and not be around to hire those better programmers in 5 years.

If you shift the question around to "should we set project and programmer standards" and "should standards improve and evolve over time", and the statement to "current standards are inadequate and irresponsible" then sure, no disagreement.

In many cases, development problems are the result of not even following what industry standards and best practices are in place now.

And one thing I don't want to see is formal programming CS programs which produce CS professors exclusively ... though some of my CS academic aquaintences dislike me saying so, from what I see, few of their graduates go into coding in industry. That's a pretty unfortunate thing for the world. To be ultimately useful, these skills need to become things which take people out of college, into industry, and successfully into challenging industry projects.

Nontrivial problems to solve. If it was easy, everyone would be doing this stuff right already. That obviously isn't true yet...

Re:nearly unlimited funding

You can't staff a whole large application development project with the best gurus: there aren't enough out there in the world.

And why aren't there?

For the same reason there aren't enough really good NFL quarterbacks for every team to have one, despite the money that is spent in trying to find them.

People differ in ability in every field; the bell curve is real, and only the people who are at the high end of the curve can be considered one of "the best gurus". They will never constitute a large percentage of the group. Ever. Furthermore, there is usually a huge difference in performance between people who are in the top 10% of their field and those who are in the top 0.1% of their field. Most people would consider those in the top 10% as "the best gurus", but really it's only that tiny segment at the very top who deserve the appelation. Even then, you can expect a marked difference between those in the to 0.1% and those in the top 0.01%. Fact of life, folks.

Re:nearly unlimited funding

What a lot of people don't realize is that being a Guru is an art.

What's the quickest way to paint the roof of the Sistine chapel? Are you going to be able to hire 30 artists with enough talent, or should you stick with the one that is qualified and a couple assistants and just wait a few years?

Can you train 30 artists to be good enough to do the work? How about 300?

After a point, being a super-coder is just as much of an art. You won't be able to produce these people, it's kinda in their soul. Great musicians pick up their first instrument and know it's what they are going to do--what they are made for. My guess would be that if you have had access to a computer for over a year and you aren't coding yet, you'll probably never be a really great coder--a real computer artist couldn't have resisted.

Hmm, maybe a better word that Guru or Architect would be Computer Artist or Code Artist? It should convey the relative rarity much better.

This should be obvious. Every other art has it's gurus, and they are usually the top 1%, 99% of the others in the field simply will never be able to do what the gurus do, regardless of training or experience. I'll never play the piano like a savant that started at age 3, period.

Rules of thumb

1. Doubling the number of coders will double the number of bugs and double the total time
   
2. Doubling the budget will double the number of coders
   
3. Extendable projects will, deadlines kill
   
4. There is no such thing as a potential bug
   
5. "Good" methods are cheap for customers, sloppy methods are profitable for companies.
   

In the end, software companies are in it for the profits. They have no lemon laws to respect, they have no trades description act to obey, no ombudsmen to answer to, no consumer rights groups to speak of, no Government-imposed standards certification and virtually no significant competition. Customers are often infinitely patient and completely ignorant of what they should be getting - the machines are like Gods and the software salesmen are their High Priests. To question is to be smote.

Were standards to be mandated - perhaps formal methods for design, OR quality certification of the end result, you would see no real impact on net software costs. Starting costs would go up, but long-term costs would go down.

Nor would you see any serious impact on variety - if anything, there is a greater range of car manufacturer and design today than there was in the 50s and 60s when cars had the unnerving habit of exploding for no apparent reason.

What you'd see is a decline in stupid bugs, a decline in bloat, an increase in modularity, possibly a reduction in latency and a move from upgrades to fix things that SHOULD have worked in the first place to enhancing things that can be relied upon to CONTINUE working fter the patches.

Money would not be made by selling the same product with a different set of defects to the same market, money would be made by always going beyond last year's horizons. The same way most manufacturers, from cars to camping gear to remote control aircraft to air conditioning units to microwave ovens to home stereo manufacturers have all been doing - very successfully - for a very long time.

The IT industry isn't going to change in the foreseeable future, the only way we'll see change in our lifetimes is if it is imposed on the Pointy Haired Bosses. We could easily see 99.9% reliable software, with no additional cost, in our homes in a year, with the lack of constant fixes actually saving money. And that's why it won't happen. Not because the IT corporations are mean, thuggish and ogreish - they are, it just isn't way it won't happen.

It won't happen because they're geared both towards the profit motive and towards the outdated notion that the market is tiny. (That last part was true - in the 1950s, when entire countries might have three or four computers in total, operating in two, maybe three different capacities. You can understand a desire to go after the after-sales service, when there simply isn't anything else left to do.)

Today, Microsoft's Windows resides on 98% of the desktop computers, but because of the support system needed to run the damn things, 98% of the world's population didn't have significant access to one. Ok, putrid green is a lousy colour, but the idea of clockwork near-indestructible laptops that - in theory - could be built to weigh 5 lbs or less and run high-end, intensive applications is beginning to filter through to the brain-dead we call politicians.

You think someone in the middle of Ethiopia who is fluent only in their native tounge is going to want to pay for telephone technical support from someone in India, in order to figure out why their machine keeps locking up?

When computing is truly available to the masses (ie: when even a long-forgotten South American tribe can reasonably gain access to one), the ONLY way it can be remotely practical is if said South American can look forward to a reliable, usable, practical experience where all usage can be inferred from first principles, and where NO software service calls are required to get things to work, ONLY required

Re:nearly unlimited funding

In fact this is the whole point - Praxis manages to deliver software with several orders of magnitude less bugs than is standard in the software industry, but does so in standard industry time frames with developer productivity (over the lifecycle of the project) on par with most non-critical software houses. Praxis does charge more - about 50% above standard software daily rates - but then when you are getting the results in the same time frame with massively less bugs a paying little extra is worth it... you'll likely save money in the support and maintenance cycle!

Jedidiah.

Re:nearly unlimited funding

At any rate - there's no such thing as bug free software. Never will be.

10 PRINK "HELLO WORLD"

Damn.

Nearly unlimited risk helps too

unlimited risk can be an incentive too.

Professor Middlebrook at caltech was an innovator in an unusual field. Sattelite electronics. Since no repairman was coming they wanted robust electronics. He desigined circuits in which any component could fail as an open or a short and it would remain in spec. You know that's a remarkable achievement if you've ever desinged a circuit before. Notably you can't really do this using SPICE. Speice will tell you what comething does but not how to design it. To do that you need a really good sense of approximations of the mathematical formula a circuit represents to see which components are coupled in which terms. And you need one more trick. The ability to put in a new element bridging any two points and quickly see how it affects the cicuit in the presence of feedback. To do that he invented the "extra element theorem" which allows you to compute this in analytic form from just a couple simple calculations. They still don't teach this in stardard courses yet. You can find it in Vorperians text book, but that's it. If you want to learn it you gotta either go to the original research articles from the 70s.

Nearly unlimited risk helps too-Theorem.

"If you want to learn it you gotta either go to the original research articles from the 70s."

http://www.edn.com/archives/1995/080395/16df4.htm [edn.com]

"The extra element theorem is used for analog circuitry. The gist of it is that you remove the reactive elements ( or dependant sources ) from a circuit and then put them back in through a process of correction factors."

[n-extra element theorem]
http://ece-www.colorado.edu/~ecen5807/course_mater ial/nEET.pdf [colorado.edu]

[Middlebrook's extra element theorem]
http://ece-www.colorado.edu/~ecen5807/course_mater ial/slidesAppC.pdf [colorado.edu]

Re:productivity around 30 LOC per day

30 LOC is net. You spend the first 45% of a high-reliability project doing the design work, and the last 45% doing the verification. The 10% in the middle is code generation.

These guys seem to be claiming they can reduce redundancy in the design work, and rework in the verification work. They're doing it by using a design-description method that prevents unambiguity (and therefore using a team that is TRAINED to write unambiguous requirements, so their magic language may not be the key), a coding method that avoids unprovable structure (and probably eliminates a lot of other sorts of flexibility), and a verification method that first validates the design and then verifies the code as it's produced (no new value there as everything has to be touched at least once anyway, and if a big bug turns up that causes a lot of code to be redone you have to redo formal verification on those units again; something that's less likely if formal verification is delayed until full-alpha code is demonstrated, having been informally verified along the way).

Their claims of massive error reduction are, at best, anecdotal. Let's see them do this after taking over a half-coded project with minimal design requirements, a hard deadline, and a budget that can be cut by governmental forces at will.

Re:productivity around 30 LOC per day

Then their ability to produce bug-free code depends, as usual, on control factors, not on real-world engineering.

In as much as a civil engineer depends on control factors via refusing customers who demand that the building have 6 stories not 4 just one month before construction is due to finish, yes. Real world engineering makes certain demands of the client. Someone who wants to build a treehouse for their kids doesn't consult an architect and a civil engineer, and civil engineers don't take contracts from people who refuse to set out some limits on what they want built, and what they expect of it.

Praxis uses solid engineering. Their "Correct by Construction" approach is solidly grounded in axiomatic mathematics and uses similar sorts of formal calculations and logical and mathematical proofs as you might expect to see from civil, electrical, aerospace, or ny other kind of engineers. Take the time to read sample chapters [praxis-his.com] from the SPARK book to get an idea of exactly what they are doing. There is very definitely quite solid engineering going on.

Re:productivity around 30 LOC per day

Their claims of massive error reduction are, at best, anecdotal. Let's see them do this after taking over a half-coded project with minimal design requirements, a hard deadline, and a budget that can be cut by governmental forces at will.

Their claims of error reduction are based on the development method and a lot of the important stuff happens very early on, taking over a half finished project that failed to follow such a method is of course not going to work. They can't make existing code bug free, but they can write new code that has vastly less errors than most software. As to hard deadlines and budgets - as far as I am aware Praxis already works with deadlines, and apparently [slashdot.org] their project for Mastercard was done on a fixed flat fee, so working with fixed or limited budgets doesn't appear to be an issue either.

Jedidiah.

No Bugs for NSA?

Uh... it's going to be kind of hard for the NSA to do its job without bugs, isn't it?

*rimshot*

Re:No Bugs for NSA?

Easier actually.

http://www.rockwellcollins.com/news/page6237.html [rockwellcollins.com]

Rockwell collins designed a secure processor for the NSA that PROVABLY can run multiple threads where each thread can get no information about the others. That way they can run processes with different clearance levels on the same CPU.

Model checking can make you big bucks if you can find the customers to pay for it.

Whatever

When you're writing software for an air traffic control system, military avionics software, or an authentication system for the NSA, the delivered code can't afford to have bugs

I've been in this industry for quite some time and let me be the first to say that I wish I could repeat this sentence with a straight face.

Here, here...

When you're writing software for an air traffic control system, military avionics software, or an authentication system for the NSA, the delivered code can't afford to have bugs

I've been in this industry for quite some time and let me be the first to say that I wish I could repeat this sentence with a straight face.

That was my first thought, particularly with military avionics. A few years ago they put out a hardware/software update for the ENS system (Enhanced Navigation System) which led to frequent crashing... and it took over a year for them to come out with a message saying that it was a bug and not to waste countless man hours trying to repair it.

It's sort of a new concept, though, as I'd never really seen such problems with traditional avionics systems (non glass-cockpit stuff). I've always attributed it to people being used to the behavior of MS Windows. And I'm not saying that to start a flamewar. I'm serious. Unreliable avionics systems should be unacceptable, but these days, that doesn't seem to be the case.

Re:Here, here...

Could you clarify here. When talking about bad guidance software for planes, "crashing" is an ambigious term ;)

Re:Here, here...

I've always attributed it to people being used to the behavior of MS Windows. And I'm not saying that to start a flamewar. I'm serious. Unreliable avionics systems should be unacceptable, but these days, that doesn't seem to be the case.

Many years ago, I remember reading a quote from an employee at a major aircraft subcontractor along the lines of "If my company paid as much attention to the quality of our work as Microsoft, airplanes would be falling out of the sky on a weekly basis, and people would accept this as normal." I've heard many people, even programmers, claim that bugfree programs are impossible to write. They are not- they just cost far more in time and money than most companies can afford in this commercial climate. When success depends largely on being first to market and bugs and crashes are accepted as a normal fact of life, then they always will be a normal fact of life.

Unfortunately, I think the blame lies at least in large part with the consumer. As long as people put up with programming errors in a $500 software suite that they would never accept in an $80 DVD player, we will continue to have these problems. Unfortunately, too many people still consider computers to be too much black magic that is out side of their (or anyone else's) grasp. Most people have little to know knowledge of how their car works under the hood, but they still believe that the engineer who designed it has enough knowledge to do it without making mistakes and expect the manufacturer to pay for those mistakes when they happen. Why should they believe any differently about the people who write the software they use?

Re:Whatever

I've been in this industry for quite some time and let me be the first to say that I wish I could repeat this sentence with a straight face.

I was pitching for "how people would like to think things are" rather than how things actually work. In practice Praxis, at least, does deliver such software, and does so with extremely low defect rates. They are proof that it can be done, even if it isn't always how things work now.

Jedidiah.

Bugs and Beta testing.

TFA cites a particular NSA biometric identification program which has "0.00" errors per KSLOC.

Now, this got me thinking. It is completely possible for a biometric identification program to identify two different individuals as the same person (like identical twins), or for it give a false negative identification (dirt on a lense, etc). Is this a bug? The code is perfect: no memory leaks, the thing never halts or crashes or segfaults, all the functions return what they should given what they are.

I think the popular definition of "bug" tends to catch too many fish, in that it seems to include all the behaviors a computer has when the user "didn't expect that output," what a more technical person might call a "misfeature." TFA outlines a working pattern to avoid coding errors, not user interface burps -- like for example, giving a yes/no result for a biometric scan, when in fact it's a question of probabilities and the operator might need to know the probabilities. Such omissions (the end user would call this a 'bug'), are solved thru good QA and beta-testing, but TFA makes no mention of either of these things, and seems to think that good coding is the art of making sure you never dereference a pointer after free()'ing it. It does mention formal specification, but that is only half the job, and alot of problems only become clear when you have the running app infront of you.

Discussion of TFA has its place, but it promises zero-defect programming, which is impossible without working with the users.

IEFBR14 is the perfect example of your point

http://en.wikipedia.org/wiki/IEFBR14 [wikipedia.org]

IEFBR14 is sort of an executable version of /dev/null. It returns immediately when called. It had four or five bug reports filed against it.

IBM could of course write a defect-free return statement. All the bugs were requirements drift that Praxis could not have prevented.

I second the motion!

I've been in this industry for quite some time and let me be the first to say that I wish I could repeat this sentence with a straight face.

I've been a controller for 13 years and have worked in the automation end of things for almost 4 years now. There is NO SUCH THING as bug-free Air Traffic Control software. The best we can hope for is heterogenous redundancy and non-simultaneous failures. Some engineers seriously think they could replace all those controllers with an intelligent algorhythm. What really scares me is that the more they try, the less engaged the people become and the harder it is for them to fall back to manual procedures when the worst happens.

Everyone used to laugh at how Windows NT could only run for 34 days before it needed a reboot. Some of our systems can't run HALF that long without needing a server switch-over or complete cold-start.

Still can have bugs

The only method I have seen with almost perfect reliability is where the inputs and outputs are overloaded to handle any datatype and can be proven mathamatically not to crash. I guess a CS degree is still usefull.

The problem is to obtain it you need to write your own libraries and not use ansi or microsoft or any other products as you can not see or trust the source code.

If you can prove through solid design and input and output types that the program wont lose control then your set. Its buffer overflows and flawed design that has not been tested with every concievable input/output that causes most serious bugs in medical and aerospace applications.

However in practice this challenge is a little unpractical when deadlines and interopability with closed source software get in the way.

Re:Still can have bugs

Did you actually bother to RTFA? Oh yeah, this is /. - stupid question. Allow me to suggest that you make an exception, and actually take the time to read TFA in this case. Praxis does not claim no bugs, they claim significantly lower defect rates than development using other methods. Which in turn means much greater confidence that the system will work as desired.

No one can ever make something that is completely "bug-free" - even in traditional, non-software disciplines. All you can do is make the probability that the system will work as high as possible. Praxis has some techniques that can help developers create software with a much higher probability of working correctly than it would otherwise have. That's a good thing, even if it doesn't result in perfection.

Its buffer overflows and flawed design that has not been tested with every concievable input/output that causes most serious bugs in medical and aerospace applications.

It's the fact that Praxis relies on static checking far beyond anything you've ever seen (using a carefully designed subset of Ada that can be statically checked in all sorts of interesting ways) that helps to ameliorate this problem, since the static check is effectively equivalent to an exhaustive input/output test.

Re:Still can have bugs

If you can prove through solid design and input and output types that the program wont lose control then your set. Its buffer overflows and flawed design that has not been tested with every concievable input/output that causes most serious bugs in medical and aerospace applications.

Praxis uses a subset of Ada together with annotations in a language called SPARK to write most of their software. They also have tools which work with such code to do considerable static checking - much as type checking catches errors, checking the annotations catches many more just as efficiently - and generate proof obligations, which they can then formally prove. That means, for many of their projects, the actaully have formal proofs that buffer overflows cannot and will not occur.

However in practice this challenge is a little unpractical when deadlines and interopability with closed source software get in the way.

Again, this is where the tools and methodology matter. Praxis delivers code as fast as traditional development techniques, so deadlines aren't the problem. They can do this by using SPARKAda and the SPARK tools to do exceptionally robust testing on a regular basis for each incremental deliverable. This allows catching bugs much earlier, when they are cheaper and faster to fix.

The only method I have seen with almost perfect reliability is where the inputs and outputs are overloaded to handle any datatype and can be proven mathamatically not to crash. I guess a CS degree is still usefull.

It is pretty much this sort of mathematical rigor, injected into the development process as early as possible, that allows Praxis to produce the sort of defect rates that they do. And yes, that does mean that developers at Praxis are probably required to have stronger math and CS backgrounds that elsewhere. Given that, due to their ability to deliver almost bug free software in very reasonable time frames, Praxis charges 50% more than the industry daily rate, yes having a math or CS degree really does count for something - more money for starters.

Jedidiah.

Uh

control structures != Turing complete. You can have loops as long as they have constant maximum bounds. Whatever it happens to be that you mean when you say "Nand is Turing complete" it makes no sense when you actually typed it. "turing-complete (for arithmetic)." makes no sense at all. WTF? Someone failed CS 315.

economics

The authors contend that there are two kinds of barriers to the adoption of best practices... First, there is often a cultural mindset or awareness barrier... Second, where the need for improvement is acknowledged and considered achievable, there are usually practical barriers to overcome such as how to acquire the necessary capability or expertise, and how to introduce the changes necessary to make the improvements.
No, the reason so much software is buggy is economics. Proprietary software vendors have to compete against other proprietary software vendors. The winners in this Darwinian struggle are the ones who release buggy software, and keep their customers on the upgrade treadmill. Users don't typically make their decisions about what software to buy based on how buggy it is, and often they can't tell how buggy it is, because they can't try it out without buying it. Some small fraction of users may go out of their way to buy less buggy software, but it's more profitable to ignore those customers.

Bugs are fine...

Luckily, bugs are just fine [washingtontimes.com] if you happen to run a company that builds voting machines, such as Diebold. And if you think that elections aren't in the same category as air traffic control, I suggest you take a tour of Iraq. Elections are very important for your continued existance upon the earth.

When bugs aren't allowed?

Ususually when the software and the phrases "life support" or "nuclear weapons" are together in the same sentence.

Not unlimited funding

The Master Money server done by Praxis was done Fixed Price, and with a warranty that says Praxis would fix any bug discovered over the net 10 years -for free-.

How many of you would be willing to place that kind of warranty on YOUR CODE?

dave (who's tried SPARK and liked it a lot, although proofs are much harder than they should be...)