Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Get HideMyAss! VPN, PC Mag's Top 10 VPNs of 2016 for 55% off for a Limited Time ×
Bug Programming Transportation

Toyota Acceleration and Embedded System Bugs 499

An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"
This discussion has been archived. No new comments can be posted.

Toyota Acceleration and Embedded System Bugs

Comments Filter:
  • Toyota: (Score:5, Funny)

    by dushkin ( 965522 ) on Saturday March 13, 2010 @01:26PM (#31464824) Homepage

    Always going forward.

    • Re: (Score:3, Funny)

      by Afforess ( 1310263 )
      "The race for quality has no finish line- so technically, it's more like a death march."

      Not my quote, so I'll give them props: http://www.despair.com/quality.html [despair.com]
      • Re: (Score:3, Insightful)

        by Z00L00K ( 682162 )

        There will always be another bug.

        And even more important - the bug may be a combination of software and hardware. Just ask what may happen if the code suddenly jumps to the wrong address. Do they use ECC memories in the electronics? What about a voltage spike? Driver has wrong socks/pants causing a spark that jumps to the OBD-II connector and messes up the CAN bus?

        If anything can go wrong - it will. Think outside the box of how bad it can be, then multiply with PI to get a value closer to reality.

        And more e

        • Re: (Score:3, Informative)

          by mmontour ( 2208 )

          And more examples of how wrong things can get can be found here: http://thedailywtf.com/ [thedailywtf.com]

          There are some good examples there, but you'll find more on comp.risks [ncl.ac.uk].

        • Re:Toyota: (Score:5, Insightful)

          by mpe ( 36238 ) on Saturday March 13, 2010 @04:40PM (#31466460)
          And even more important - the bug may be a combination of software and hardware. Just ask what may happen if the code suddenly jumps to the wrong address. Do they use ECC memories in the electronics? What about a voltage spike? Driver has wrong socks/pants causing a spark that jumps to the OBD-II connector and messes up the CAN bus?

          Other questions would be "What kind of transducer is measuring the input?"; "How many transducers are there?" and "What output do you get in the case of a failure?"
          Note that there are applications where an unknown throttle setting resulting in full power being applied is the right thing to do. Maybe Toyota through they were building a light aircraft rather than a car...
  • by Darkness404 ( 1287218 ) on Saturday March 13, 2010 @01:28PM (#31464840)
    Most software is nearly -impossible- to test under flawless conditions. Especially embedded systems with small amounts of CPU power and memory.

    Plus, all this hype around these Toyota acceleration problems is just that, hype.
    • by Anonymous Coward on Saturday March 13, 2010 @01:33PM (#31464870)

      Right, just hype. Except for those families were killed by the Toyota acceleration problems.

      • by Darkness404 ( 1287218 ) on Saturday March 13, 2010 @01:45PM (#31464958)
        ...And if you look at the facts, you can see that all of the symptoms could easily be caused by driver error. Look at this http://www.nytimes.com/2010/03/11/opinion/11schmidt.html?scp=1&sq=driver%20error&st=cse [nytimes.com] (currently the page doesn't need registration, your results may change in the coming days/hours).
      • by zappepcs ( 820751 ) on Saturday March 13, 2010 @02:03PM (#31465108) Journal

        There are a couple of things that should be mentioned here. NASA has shown what it takes to make very small, very good code. Sure, they too have failures, but 'nearly' bug free code is quite expensive. Second, writing code is not quite like trying to create a hand crafted dashboard, if the dashboard fades, no one dies. Embedded software is quite a different beast from your normal desktop applications. When you add motion control and interaction with the code, it difference between them gets even more complex. Software in vehicles should be two things:

        Open - let lots of folk see what could be wrong
        Audited - audited to meet specific standards of safety and operation. Not quite the self-defeating government regulations, but more of a case by case issue: if the software has control or input to the control mechanism for the engine, braking system, suspension etc. it must meet minimum standard testing requirements. Any action that _could_ arbitrarily apply mechanical action must be tested and controlled beyond all reasonable testing/doubt. Everything should be tested, down to a pet chewing on the control cable harness.

        Consumers are encouraged to think the vehicles they buy are safe and require no special knowledge of engineering or mechanics to operate. As long as they are given to think that, then passenger vehicles should be made to be just this way.

        The problem for Toyota now is multifaceted. One, they have a PR shitstorm to deal with. Two, there is a dollar effect of this problem. Three, it's now on the shoulders of Toyota to get this part right for the rest of the passenger vehicle making industry.

        It's possible that they might walk away from this fire with only minor long term burns and the reputation for building the safest vehicles. BUT, reading the article of this post and paying attention while doing so is necessary... IMO

        • Re: (Score:3, Interesting)

          by cynyr ( 703126 )

          I'm still failing to see how the cars got locked in gear? every car i have driven has allowed the driver to shift the car into neutral regardless of everything else. This is both in automatics and definitely in my manual transmission cars(does anyone make a drive by wire clutch, outside of performance/race cars?) I fail to see why this is a huge issue that needs to be solved in the next 10 minutes and be 100%? how is a sticky peddle (software or otherwise) any different from the throttle body getting stuck

          • Re: (Score:3, Insightful)

            by zappepcs ( 820751 )

            The "PR shitstorm" is way way over-hyped, it would be simple for the news to simply state "Toyota has confirmed an issue effecting the engine speed controls, and have issued a recall. If this happens to you while driving Toyota advises drivers to shift the car into neutral and engage the 4 ways and pull over in a safe location. If your car has a push button start be aware that you will need to hold it down for up to 5 seconds to shut down the engine." The fact that some people have died as a result of poor

    • by DeadPixels ( 1391907 ) on Saturday March 13, 2010 @01:33PM (#31464872)
      By and large, it would seem that Toyota should probably be looking for exceptional conditions rather than typical ones. Correct me if I'm wrong, but if a relatively small number of vehicles have actually exhibited the acceleration issue, it would seem like any bugs related to that would be in conditions that may not occur very often during typical driving. Seems to me that "outlier" cases or unusual methods of testing would be the best way to start; testing with typical driving conditions might not show anything.
    • Nothing is impossible. It's simply a matter of cost vs. benefit. Coding is many more lines today than it was 15 years ago. And a lot of copy paste of libraries you haven't coded or tested yourself. So ofcourse problems occur. But testing SHOULD be more than simply compiling the code.

      Back in the day when I took my masters in computer science, we were forced to show how we tested EVERY loop, if, while or orhter "choice" or iteration of a possiblity. Simply to ensure that the code was 100% stable. Using closed

      • Re: (Score:2, Insightful)

        Programmers today just give up too easily, and companies cut costs by scriping on testing, ducumentation and verification.

        I'll start with the cheap joke - apparently you didn't learn the lesson too well.

        And then the actual point - this is the real world, sir. Academics may have the luxury of taking as much time as they like to never produce result. That's not how things actually work for the rest of us.

    • by goombah99 ( 560566 ) on Saturday March 13, 2010 @01:55PM (#31465044)

      I'm loving this conversation here because I've gotten crucified in slashdot before for making simmilar comments to the whole thread here. I grew up in a family of top managers of Boeing systems engineers. They hated computers. My dad never even learned how to turn one on. He hired other monkey to use the computers. As A child I was regailed with wonderful stories of every hard lesson in safety my dad had learned over his lifetime. He loved world war II because they got to use cutting edge designs for balls out performance yet at the same time learned how to make things reliable by disecting the accident. He would tell me about the accident that taught them that the engine pumps need to be at full speed but flow stalled on take off so that there's no lag when you hot swap after a pump fails. He told me of the accident where they learned not to route 100% of the control system wiring through any one junction box. etc...

      Probably because of all these hard won lessons boeing for years insisted on fully mechanical or hydraulic flight surface controls. Whereas Airbus and other jumped on the fly-by-wire concept early. My dad would spit after hearing some youg person tout all the advantages of fly by wire. He knew them perfectly well. He was big on accepting new innovations to reduce fuel costs and increas performance. He was not a luddite. But he had a safety background that told him these electonic systems were hard as hell to validate and hard as hell to make truly independent from each other.

      For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it. From what I've read it's now suspected that the latest airbus crash in the pacific had one of it's root problem in the voting nexus where a superior computer over ruled a more primitive safety system.

      While we all know that computer software validation is hard if not impossible. It's not something we readily admit here on slash dot. It's because for years people like my dad would throttle the innovations the computer engineeers wanted to implement. I think as a result there became this culture of computer engineers that presented the case that embedded computing could be made safer than it really could be to offset that.

      So now we come full circle and have to admit there is this middle ground. Just because a computer can improve perfromance does not mean it's reliable and safe. The old guys had a point after all when it came to safety.

      Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft). In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash. Conversely, composite wings on UAVs allow them to absorb a lot of bullet holes with no loss of control and to operate under higher perfromance conditions.

      The point is that safety and performance are trade offs when both are pushed to the limit. The old guys know a lot more about safety than you might expect. The young guys are all about performance.

      • For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it.

        Sounds a little like Minority Report [wikipedia.org], doesn't it?

        Each of the three precogs generates its own report or prediction. The reports of all the precogs are analyzed by a computer and, if these reports differ from one another, the computer identifies the two reports with the greatest overlap and produces a majority report, taking this as the accurate prediction of the future.

      • by roman_mir ( 125474 ) on Saturday March 13, 2010 @02:23PM (#31465284) Homepage Journal

        A year ago I was watching one of Discovery programs I think and they had a couple of guys who supposedly implemented a piece of software, that would allow an airplane to fly and land safely if for some reason, while in the air, the tale would brake off or rudder would just stop working. They relied on a fly by wire airplane of-course and controlled the yaw with all other surfaces by applying very slight changes to the motion. They were saying a human could do this if extremely lucky, but software was able to do it almost always.

        Just something to think about.

        • Re: (Score:3, Insightful)

          by b4dc0d3r ( 1268512 )

          Justifying something based on its performance on outlier conditions has never been a good idea. The number of accidents where the tail falls off is probably greater than the number of accidents where the tail falls off and the software would be able to compensate.

          Maybe having the "oh shit" button turn this on would be a good idea, but I think if you look at the number of crashes and their causes, you'd want to build redundancy in the rudder or strengthen the tail.

        • Re: (Score:3, Interesting)

          by X0563511 ( 793323 )

          I have an example.

          In a simulator (yes, a simulator) I was flying a VTOL type aircraft. Pulled a turn at too great a speed and broke off a few control surfaces. Maddening spin, completely unrecoverable (at least for me).

          Tapped the button to enable "artificial stabilization" - which in this craft, enabled "puffers" charged with compressed air (driven by the engines, which still worked) - the computer control algorithms managed to use the remaining control surfaces and these puffers to level the craft and redu

      • by Beryllium Sphere(tm) ( 193358 ) on Saturday March 13, 2010 @02:38PM (#31465382) Homepage Journal

        I worked on an embedded flight system there, and deeply respected people like your dad.

        Boeing works under the eye of a certification authority who has to approve the safety of a design including, at least in the system I worked on, human factors. If there's anything comparable for cars, I haven't heard of it.

        Boeing would not have made a pilot have to guess at how to turn an engine off (people with older cars, it's no longer a matter of turning a key).

        Inputs were checked for consistency and validity. The specs would have anticipated what to do if the accelerator and brake were both full on at the same time.

        There was a culture of worst-case planning and redundancy.

        Also, if Boeing built a car, it would have a flight data recorder which investigators could examine and say for example "Looks like both(*) potentiometers on the accelerator went hard over at the same time, so we go look on the branches of the fault tree where there's a common-mode failure in the potentiometers or the pedal is down due to mechanical or pilot error".

        (*) If I remember correctly from my obsessive pre-purchase research on Priuses, there are two separate sensors for accelerator position.

      • by timeOday ( 582209 ) on Saturday March 13, 2010 @03:18PM (#31465700)
        You make it sound as computers killed a golden era of safe engineering, as if perhaps we should go back. Let's keep some perspective here. Were cars back then simple, predictable, and solid? Perhaps. They were also death traps. How do the deaths with a suspected link to throttle software stack up statistically to those saved by anti-lock brakes? How does Toyota safety - on any model you choose - stack up to the competition when measured in deaths per millions of miles? I doubt this problem is even enough to appear in those big-picture statistics.
        • Re: (Score:3, Insightful)

          by jamesh ( 87723 )

          Amen. Fly by wire can never be perfectly safe - no matter how well a system is designed it can still fail. As long as its safer than the mechanical systems we're still ahead though.

      • by DerekLyons ( 302214 ) <{moc.liamg} {ta} {retawriaf}> on Saturday March 13, 2010 @03:20PM (#31465716) Homepage

        Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft).

        This one comment makes me wonder about the veracity of the balance of your account.

        1. Metal fatigue was known about long before the Comet took wing in 1949.
        2. The fatigue cracks on the Comet's occurred at skin penetrations (windows and hatches).

        In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash.

        Then there is crunchy bit of FUD, which fails to mention that more than a few of those accidents are also associated with extreme control surface movements (inducing extreme stresses) prior to the failure.

        • Re: (Score:3, Informative)

          by goombah99 ( 560566 )

          Derek, as you might have noticed I was keeping it short on the comet disaster. But to expand. yes of course metal fatigue as a phenomena was known before the Comet disaster. What my Dad told me was they learned that they did not know about how to design for it yet. They did not have any computer modeling to know what flight stress really did to winds and to metal. they hardly had any way to measure material strength changes in-place. The people who built the Comet we no dummies so clearly they discove

          • by DerekLyons ( 302214 ) <{moc.liamg} {ta} {retawriaf}> on Saturday March 13, 2010 @07:00PM (#31467530) Homepage

            Derek, as you might have noticed I was keeping it short on the comet disaster.

            One can be short without being wrong. You were both short *and* wrong.

            The people who built the Comet we no dummies so clearly they discovered a scaling issue in metal no one had encountered before nor new how to design for at that time.

            No, actually they discovered (as is widely documented in aviation histories) that they failed to correctly account for the stresses caused by multiple pressurization and depressurization cycles. They knew perfectly well how to design for metal fatigue, but lacked information on how that fatigue would manifest itself.

            The point I was making was that just because you can't foresee a problem in something new does not mean you cant anticipate there might be problems you can't foresee. The switch to composites opened up the same sort of issues that the comet did.

            I merely pointed out how you have the story wrong, not that your point was false.

    • "Impossible to test", but that does not mean that it's impossible to write bug-free software. It requires a substantially different approach to specification and construction than most people/companies currently use. Model Checking (http://en.wikipedia.org/wiki/Model_checking) and SPARK (http://en.wikipedia.org/wiki/SPARK_(programming_language) ) are two approaches that work. It's worth looking at what the commercial avionics industry requires for its embedded software, where 10 ^ -9 is the requirement f

    • by Loconut1389 ( 455297 ) on Saturday March 13, 2010 @03:05PM (#31465604)

      Indeed. I've done some embedded work myself. I wrote a power supply controller that used DACs to trim the voltage using some analog control ports on the DC to DC converter modules- it also monitored the PowerGood lines on the DC:DC's and linears and was programmed to shut down if one deasserted without a prior command telling it to do so. It had an I2C control network that could request status of bunch of aspects of the board including temperature, voltages, etc. Not wanting to risk blowing out a $10k FPGA with a $4.00 MCU, we had test boards with no FPGA on and some with cheaper FPGAs, and I also had a dev kit with the board on it hooked to a logic analyzer so we could emulate all sorts different scenarios and hopefully protect the FPGAs. Ultimately, a few problems emerged. With a particular combination of testing apparatus and polling rate, the I2C would receive interference and miss or corrupt some data. It was almost impossible to replicate reliably. This in turn exposed an oversight/bug where because of the skipped (as far as the power supply MCU was concerned) bytes, the wrong DAC values were being written, overvolting or undervolting the supplies- but it really only surfaced on the fully populated boards. This lead to a change in the I2C wiring/termination and a move to a keyed and transactional approach that required writing a key value to an address, writing the new data, then optionally reading back the data again, and lastly writing another key to a different address to either commit or roll back. Point is exactly what the parent said, it's very difficult to test some of these things because the problems may be an unusual chain of events or due to very specific circumstance in what's hooked to what and how much power is being drawn in the circuit at the time, etc.

      The other portions of the code that performed monitoring and emergency shutdown caught the overvoltages very quickly and shutdown the FPGA in the span of a couple clocks. In the end we only lost one board, and it was due to ESD despite using proper handling techniques and equipment.

  • Infallible fail. (Score:5, Insightful)

    by jeckled ( 1716002 ) on Saturday March 13, 2010 @01:32PM (#31464856)
    Drive by wire is great and all, but I'd feel much better with a physical fail-safe than their "infallible" software. I am aware of the physical remedies for the issue, but I'd like to see the brake pedal override the accelerator.
    • Re:Infallible fail. (Score:4, Informative)

      by shrimppesto ( 766285 ) on Saturday March 13, 2010 @02:09PM (#31465176)

      i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...

      • Re:Infallible fail. (Score:4, Informative)

        by timeOday ( 582209 ) on Saturday March 13, 2010 @03:25PM (#31465756)
        You could be referring specifically to last week's incident [aol.com], which I found fishy from the start:

        Skeptics of Sikes also cite the 911 tape that was released shortly after the incident. During the tape, the dispatcher repeatedly told Sikes to put the car in neutral in order to stop it from accelerating. Sikes did not comply with her instructions or the instructions of the officer on the scene who told him to do the same thing via his public address system as they tore down the highway.... Sikes claimed he thought that would "flip the car."

        Beyond the call itself, the Associated Press reports that Sikes's car was equipped with a brake override system, something that should have slowed the car down once he stomped on the brake pedal.

      • Re: (Score:3, Interesting)

        by jc42 ( 318812 )

        i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, ...

        Except we have testimony from any number of the Toyota acceleration victims that they had put the transmission into the "N" position, but the car just ignored it and kept accelerating. They also claimed that they knew how to use the brake, but the car also ignored that.

        As a software guy, I'm quite familiar with ways that software will do things like this, and I find th

  • by homer_s ( 799572 ) on Saturday March 13, 2010 @01:33PM (#31464868)
    From here [washingtonexaminer.com]:

    In the 24 cases where driver age was reported or readily inferred, the drivers included those of the ages 60, 61, 63, 66, 68, 71, 72, 72, 77, 79, 83, 85, 89—and I’m leaving out the son whose age wasn’t identified, but whose 94-year-old father died as a passenger.

    These “electronic defects” apparently discriminate against the elderly, just as the sudden acceleration of Audis and GM autos did before them. (If computers are going to discriminate against anyone, they should be picking on the young, who are more likely to take up arms against the rise of the machines and future Terminators).

    Some more data here [theatlantic.com]

    • by maxume ( 22995 ) on Saturday March 13, 2010 @01:35PM (#31464890)

      Be careful to note that the 24 cases discussed there are only the ones that have led to serious incidents.

      • Would that suggest a potential correlation between reaction time and general seriousness of the possible incidents?

        • by maxume ( 22995 ) on Saturday March 13, 2010 @02:19PM (#31465260)

          To me it suggests that older drivers are having more difficulty coping with the situation once it arises.

          Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:

          http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes [forbes.com]

          (They link the 911 recording:

          http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/ [thetruthaboutcars.com]


          So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.

          But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).

          • by causality ( 777677 ) on Saturday March 13, 2010 @02:43PM (#31465432)

            To me it suggests that older drivers are having more difficulty coping with the situation once it arises.

            Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:

            That part is strange. Uncontrolled acceleration is a much greater risk to life and limb than the red-lined/blown engine you might get if it were put into neutral with the throttle wide open. Being "afraid to try neutral" makes no sense.

            They link the 911 recording

            Just an irrelevant side note: I've always found it low-class and tacky that phone calls made to 911 become publically available, especially when you hear them on the news. The message is, "hey sir or madam, remember that moment when you were highly emotional and had no idea if you were going to live or die? Well, we've got great news! That highly personal moment of reflection on your own mortality is now a public spectacle for millions of people! It's okay, we make a profit from this! No we won't share that profit with you..."

            I realize it's a government service funded by taxpayer dollars. That explains how this is possible. It fails to explain how this is the best or most honorable thing to do.

            So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.

            That part generally shouldn't be a surprise. I'd imagine it also helps if you can keep calm and avoid panicking, as panicky people often fail at things they could do easily if they were not in a state of deer-in-headlights fear.

            But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).

            Nor does it explain why older drivers were disproportionately affected. Possibly the Toyota brand is more popular among older drivers because it historically has retained a decent resale value. While nothing the driver does should ever cause this kind of uncontrollable automatic acceleration, perhaps older drivers tend to have habits that somehow manifest whatever the actual underlying problem is. There are a lot of coincidences and correlations being pointed out in this discussion but unfortunately there seems to be little certainty about whether they are more than that.

    • by WrongSizeGlass ( 838941 ) on Saturday March 13, 2010 @01:49PM (#31464994)
      Now, come on. We can't blame age for this. I plan to be that old one day ... as long as I don't buy a Toyota, that is.
    • Re: (Score:2, Informative)

      by f16c ( 13581 )

      Something to understand about those statistics: This is a self selected group based largely on income. Camrys may be everywhere but Prius' tend to be expensive.

      • by Lehk228 ( 705449 )
        also, since this is the group of serious accidents, older people are less likely to react correctly and in a timely manner to the problem.
      • 2010 Camry base price: $19,595 - most expensive trim starts at $26,400
        2010 Prius base price: $22,800 - most expensive trim starts at $28,070

        Even a fully optioned Prius couldn't be described as expensive by new car standards. The price difference isn't enough to expect it would impact the type of owners very much... they fall in the same price range.

  • Testing. (Score:5, Insightful)

    by Ihlosi ( 895663 ) on Saturday March 13, 2010 @01:34PM (#31464884)
    Testing only confirms the absence of known bugs. Never forget that.
  • Logic of Testing (Score:5, Insightful)

    by Renegade Lisp ( 315687 ) * on Saturday March 13, 2010 @01:38PM (#31464908)

    David Cummings does seem to know what he's talking about, but as it is written, there is some strange logic in the article.

    If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying.

    Testing cannot prove the absence of bugs, only their presence. There are two things that do not follow from this:

    • If you don't find any bugs, then your software doesn't have any.
    • If you don't find any bugs, then there must be some left in your software.

    It sounds to me as if Toyota is saying the former, while Cummings says the latter. Neither is a correct conclusion.

    • by marcansoft ( 727665 ) <hector@marcanso[ ]com ['ft.' in gap]> on Saturday March 13, 2010 @01:43PM (#31464944) Homepage

      Given practical software engineering conditions though, a) is highly unlikely while b) is highly likely.

      • by Renegade Lisp ( 315687 ) * on Saturday March 13, 2010 @01:52PM (#31465018)

        I'll grant you that, but what I don't understand is this:

        If you test, and do find some bugs, does that allow you to put any more trust in your software than if you tested and didn't find any?

        • That's a tricky question. Testing and finding bugs ought to allow you to put more trust in your testing methodology, which subsequently can increase your trust in your software once you stop finding bugs. Testing and finding no bugs, hard as you try, quite likely means you aren't trying hard enough. Very rarely will software reach a ceiling of reasonable test-proofness before being shipped that cannot be improved with subsequent, more dedicated, more specific testing after some issues are detected in the fi

    • How about this howler:

      The only viable theory we could come up with was that an interrupt (an external hardware stimulus such as a timer going off) had occurred at just the right microsecond within the execution of Stolper's software. Furthermore, we theorized, the operating system (the equivalent of Windows on the flight computer) had a bug that caused it to misremember whether an arithmetic carry had occurred just before the interrupt. Although highly unlikely, it was the only credible explanation we could

      • So they speculate that code they wrote had an interrupt routine that was not bracketed with PUSHF/POPF instructions!!! Which is like Assembly 101.

        I didn't read it that way. He's talking about an arithmetic carry condition being misremembered across an interrupt. This sounds to me like a CPU-internal hardware condition that might not have been included in the regular set of data that you save across an interrupt. Maybe the wrong type of PUSHF/POPF was used. It certainly doesn't sound like Assembly 101.

  • I'm in the market for a car and everyone is picking on Toyota now.

    I don't believe in stupid media hype. I don't believe cars rewire themselves. And I know how to hit the breaks, shift into neutral, and/or turn off the key when I want the car to go slower (so far, hitting the breaks has always proven adequate).

    Are there any really good deals on Toyotas available?

    • by thewils ( 463314 )

      Be careful about switching off the ignition while you're still moving. It could cause loss of steering and/or braking ability. That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.

      • Not quite true. You may lose "power assisted" braking and steering but the wheel will still steer the car and the brakes will still work, it will just take a little more effort. Those old enough will remember a time before power assisted steering and braking.

        • If they turn the key all the way off the steering wheel can lock. At least it does on my car.
          • Re: (Score:3, Informative)

            by MachDelta ( 704883 )

            Any vehicle built in the last thirty or fourty years will not allow the steering column to lock unless the transmission is in park. If you're in drive (or neutral) you can only turn it to "off", not all the way to "lock". This was to prevent an errant knee from locking the steering while you're doing 70 on the freeway. Happened to me once, except I was only doing 45 on a bumpy ass gravel road when my knee smacked into my keychain. It was startling, but not particularly dangerous.

      • by Kohath ( 38547 )

        Yeah, it has never actually been necessary to turn off the ignition. But my engine has killed while moving before. Steering and brakes still function, not as well, but good enough for a safe stop.

        Software failed to fix my spelling of "brakes"! Someone call David Cummings!

      • That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.

        Lack of one leads to the other. I'm sure he'll figure it out ... I hope.

        Since the actual problem hasn't been identified yet, who knows if all Toyotas have it or just a select few? Maybe it's a date/time bug? Integer overflow bug? I don't know what the problem is, but I know where it is and I'm not willing to put my family at risk in order to secure a deal on a car that may be more dangerous than most.

      • Re: (Score:3, Interesting)

        by bitingduck ( 810730 )

        I used to have a car where the engine would suddenly turn off for no reason while driving, often at exciting moments like getting onto the freeway. It was pretty easy to put it into neutral (it was an automatic), turn the key to "acc" and try to restart the engine (usually with success) without accidentally locking the steering wheel.

        It went on for some time until I convinced the repair guys to clean all the electrical connections from the computer to the fuel pump. The car had lived most of it's life in

    • by John Hasler ( 414242 ) on Saturday March 13, 2010 @02:19PM (#31465262) Homepage

      > And I know how to hit the brakes...

      With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine (this is, IMHO, a fundamental design defect).

      > ...shift into neutral...

      The computer may not let you do that with the car moving and the engine at high rpm. After all, the engine and/or transmission might be damaged (another design defect).

      > ...and/or turn off the key...

      Some of these vehicles don't have keys: just a radio remote. The emergency shutdown procedure is to hold a button down for three seconds (another design defect).

  • It seems he wants more software. This time to check for both pedals being depressed at the same time. One more thing to break.

    The less these embedded systems have to do, the better.

  • Didn't we already "solve" this problem in the airline industry by breaking down larger more complex software into separate physical components that can be more easily verified? Can we do the same with cars?

    Just split one computer into half a dozen much smaller, simpler, little units and set the valid IO conditions for them, then have the components around them sanity check their output.

  • Pardon my laziness for not investigating this myself, but doesn't the Prius have a mechanical (hydraulic) link to the brakes that engages when the pedal is pushed down far enough? I realize that the first portion of the braking is done electronically (for the regenerative braking system), but in an emergency wouldn't a full application of the brakes slow down the vehicle?

  • I've said time and time again, "Never replace hardware with software" because
    something dedicated to the task will always work better, or be less failure
    prone (more often than not).

    Would Toyota be having these problems with an accelerator cable vs electronic?

    99% sure the answer is "no"...heck the solution is add some grease, make sure
    it isn't pinched/looped too tightly and/or add tension to the pedal side.

    Or, replace the damn cable with a new one...a 20 to 30 minute task.
    (less than 10min on a motorcycle)


    • by DerekLyons ( 302214 ) <{moc.liamg} {ta} {retawriaf}> on Saturday March 13, 2010 @03:32PM (#31465794) Homepage

      I've said time and time again, "Never replace hardware with software" because
      something dedicated to the task will always work better, or be less failure
      prone (more often than not).

      On the flip side, the USN replaced complicated and heavy hardware analog computing systems for [SSBN] missile guidance systems with software running on a digital computer, and MTBF went through the roof and maintenance man hours and MTTR through the floor. The same thing happened when they replaced the analog torpedo fire controls with digital ones. The same thing happened again when the hovering system controls were upgraded to digital.
      Now, before you claim that is a limited set of examples, I invite you to consider the millions of incident free flight hours accumulated by fly-by-wire aircraft. Or the replacement of DIP switches in PC's with software configuration. Etc... Etc...

    • Re: (Score:3, Interesting)

      by John Hasler ( 414242 )

      > Would Toyota be having these problems with an accelerator cable vs
      > electronic?

      GM once had a very similar problem with a 70s car with a cable. An engine mount failure would allow the engine to rotate under acceleration in such a way as to yank the cable to full throttle and then jam it, causing the car to run away. The resulting collision would knock the cable free and as collisions often break engine mounts, the evidence disappeared.

      Computerized systems are usually more reliable than mechanical o

  • On a side note, our legal system forces us all to lie.
  • Did Toyota forget to convert imperial to metric unit like those Mars Pathfinder?
  • prove a negative? (Score:3, Insightful)

    by sjs132 ( 631745 ) on Saturday March 13, 2010 @02:02PM (#31465096) Homepage Journal

    Isn't this like proving God doesn't exist?

    They can test and test and not get a result that said this is the bug, so they assume that it doesn't exist.

  • the acceleration was caused by some external EMI event? Why?. It is well known amongst ham radio operators that some cars do not have sufficient EMI protection so that an otherwise correctly installed transceiver will interfere with the car electronics when used. Pure speculation I know......
  • by dbc ( 135354 ) on Saturday March 13, 2010 @03:00PM (#31465564)

    ... why does everyone assume it is a software bug? I agree that it very well could be an undiscovered software bug. But there are so many more sources of erroneous behavior in an embedded system that *even* *if* the software were flawless (ummm... just go with me a minute... :) an automotive environment can cause all manner of strange glitches. I work with robots, lots of DC motors causing commutation noise on the power supply, long (several inch) distances between units that must talk to each other and therefore may have a different opinion as to ground reference voltage... many things can get wacky. Even flawless code needs a watchdog timer to get you out of weird states that power glitches that put you into. Power supply spikes can cause the program counter to jump to very odd places, with odd, corrupted stuff in RAM. Ground level shifting can cause communication glitches. CAN bus is *extremely* robust, so bad data should not get through... but what does get through? Does the system as a whole get into a weird state if packets drop?

  • by descubes ( 35093 ) on Saturday March 13, 2010 @03:44PM (#31465940) Homepage

    Several years ago, I designed the software for a real-time automotive test system called HP ECUTEST (I think the official name was HP Design Span DS5470, but let's not waste time on HP's cold dead fish naming conventions). It simulated a car from an electric point of view. You connected an electronic control unit (ECU), and it had basically no way to tell it was not in a real car. Think of it as The Matrix for car electronics.

    One of our first customers wanted us to test it with a reliable, proven, tested, tried and true ECU, something that was on the road in cars for several years already. So we did. And I noticed something odd. The ECU worked fine when we "drove" a car normally, but at idle, it would basically slow down, one RPM at a time, until it stopped. However, if I changed the value of the input corresponding to the accelerator pedal, it would reset the idle speed to the default, something like 800rpm.

    Finally, after eliminating the possible bugs on our side, we tell the customer. Their first reaction was "no way". But after a week and a demo of the problem, they finally made a connection. They had this elusive bug of some car customers complaining that their car would sometimes stop when idle. It turns out that in a real car, chassis vibrations generally caused minute changes in the input value for the accelerator. So the ECU would correctly recompute its idle speed. However, if there was no change, like if the pedal was more rigid than usual, the bug would trigger.

    The root cause was a routine that wanted to optimize idle speed to be as low as possible, but for some reason kept cached data if the accelerator had not changed, so it thought the engine was still running smoothly.

    We found such bugs in practically all ECUs we tested for the first time. The most impressive one was in a V8 ECU that was basically a V8 until 1200rpm, then a V7, then a V6, and basically a V2 above 4000 rpm. The customer had hoped we'd find something, because they didn't get all the power they expected from the engine. Obviously. It was hard to find without our system, because the injectors that fired were differnt from cycle to cycle, so more simple instrumentation saw all cylinders running. The root cause here was that the software badly exceeded its real-time envelope... Ouch.

  • by RobinH ( 124750 ) on Saturday March 13, 2010 @05:30PM (#31466836) Homepage

    I do industrial automation for a living, and machine guarding/safety is a major component of the job. There are now, in the last few years, software based safety products that are provably just as safe as a hardware only safety products. The key is that it's not just about rigorous testing, it's about correct design. If you want category 4 protection, you need to be sure that:

    1. No single component failure can leave the system in an unsafe state
    2. The component failure is detected
    3. The system cannot be restarted without correcting the failure

    Software becomes another component. Therefore you need to have redundancy in your software. Government regulators that certify these safety systems as compliant want to see you prove that a single component (i.e. unit of software) can't malfunction and leave the system in an unsafe state. What a lot of companies do is they have two independent processors each monitoring the inputs to the system in parallel, and each generating the required outputs. The processors are typically sourced from different companies, and the circuit boards are designed by different teams. The software running on each processor is written by a different team. If both processors agree on the outputs, the system drives those outputs, and if not, all power is dropped to everything and the system can't be restarted (may need to be replaced, etc.).

    Those of us in the industry were skeptical of software based safety at first, but given the above facts and a decent amount of regulatory oversight, I'm satisfied that it will live up to the design criteria. That doesn't mean an error can't happen, but it makes the probability low enough that we can live with it.

    The latest thing is safety systems running their I/O across networks like DeviceNet and even Ethernet/IP (the IP stands for Industrial Protocol, not Internet Protocol). Again, I was at first skeptical, but they use a protocol layering on top of the network using timestamps and redundant processors on both ends with reasonable failure modes that the system is provably safe, within reasonable limits.

    So you can make safe embedded systems, but without being able to inspect the design and see that it lives up to these guidelines, Toyota can't ever *prove* that the system is safe.

  • by Stormy Dragon ( 800799 ) on Saturday March 13, 2010 @05:46PM (#31466956) Homepage

    The Times also helpfully provides a list of all the people who have died in "sudden acceleration" accidents involving Toyotas:

    Toyotas, deaths and sudden acceleration [latimes.com]

    If you look through the list at the ages mentioned, one begins to notice a rather odd pattern: 18, 21, 32, 34, 44, 45, 47, 56, 57, 58, 60, 61, 63, 66, 68, 72, 72, 77, 79, 83, 85, 89

    This is a most peculiar bug indeed in that it seems occur primarily when the driver is elderly. Or perhaps, as with previous "sudden acceleration" scares, this will ultimately turn out to be the result of people slamming on the gas when they menat to slam on the brake and then trying to blame the car for their error.

Say "twenty-three-skiddoo" to logout.