Toyota Acceleration and Embedded System Bugs 499
An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"
Toyota: (Score:5, Funny)
Always going forward.
Re: (Score:3, Funny)
Not my quote, so I'll give them props: http://www.despair.com/quality.html [despair.com]
Re: (Score:3, Insightful)
There will always be another bug.
And even more important - the bug may be a combination of software and hardware. Just ask what may happen if the code suddenly jumps to the wrong address. Do they use ECC memories in the electronics? What about a voltage spike? Driver has wrong socks/pants causing a spark that jumps to the OBD-II connector and messes up the CAN bus?
If anything can go wrong - it will. Think outside the box of how bad it can be, then multiply with PI to get a value closer to reality.
And more e
Re: (Score:3, Informative)
And more examples of how wrong things can get can be found here: http://thedailywtf.com/ [thedailywtf.com]
There are some good examples there, but you'll find more on comp.risks [ncl.ac.uk].
Re:Toyota: (Score:5, Insightful)
Other questions would be "What kind of transducer is measuring the input?"; "How many transducers are there?" and "What output do you get in the case of a failure?"
Note that there are applications where an unknown throttle setting resulting in full power being applied is the right thing to do. Maybe Toyota through they were building a light aircraft rather than a car...
Re:Toyota: (Score:4, Funny)
Impossible to test (Score:5, Informative)
Plus, all this hype around these Toyota acceleration problems is just that, hype.
Re:Impossible to test (Score:5, Insightful)
Right, just hype. Except for those families were killed by the Toyota acceleration problems.
Re:Impossible to test (Score:5, Informative)
Re:Impossible to test (Score:5, Insightful)
"Unacceptable" is strong. Sad, yes, but this is real life. There is no such thing as zero risk. Taking the attitude that it is somehow achievable despite the utter impossibility is something that makes for a good trial lawyer but a terrible human.
Re:Impossible to test (Score:5, Insightful)
The is a component of moral empowerment though, you have to consider. Most people are more willing to accept risk if they control the situation, even if the risk is greater. Other people are more accepting of an inherent justice of in the results when something bad happens to someone else who they feel was in control of the situation than when they were not.
Consider on a per person per mile of travel basis a drunk "walker" is more likely to cause a traffic related fatality than a drunk driver. They do things like stumble off sidewalks into traffic, misjudge the rate of on cumming traffic and run out into busy highways, sit an take a rests on unlit rural roads and more. Still we vilify the drunk driver because when they cause a traffic fatality chances are they are not the individual contributing to the statistic, where as with the walkers they are usually the one killed.
If we really minimizing risk we would be more condemning of drunk walking than driving because someone is more likely to die. We don't operate that way though, we don't think that way. Many people would take a friends keys, few would forcibly restrain them if they could not be convinced to stay a little longer and sober up, even though that friend would be safer behind the wheel.
The same thing applies, most of us would be more willing to accept our loved one died because they were not able to control a set of mechanical and hydrolic linkages correctly and quickly enough to avoid and auto accident than we are when a software system fails to do the same, even though the later was far less likely.
I am not saying that makes sense in moral terms, statistical terms, or anything. In fact the more objectively you look at it the less sense it makes to not use drive by wire and computerized systems but "we" still don't "feel" that way about it.
Re: (Score:3, Informative)
Except that drive-by-wire systems have absolutely no possibility of making the operation of the car safer (more efficient, sure, but that's it) and many many ways in which to make it less safe, so you're wrong about it making more sense to use drive-by-wire. I can't figure out what you're thinking with that statement.
That's flat-out not true. Traction Control and brakes and power synchronizing together (which is a lot of what drive-by-wire DOES) absolutely, in most cases, makes the average driver safer. Personally, I'm inclined to think that as drive-by-wire improves (until eventually, hopefully, the really dangerous part (i.e. the driver) will go away), we will in general progress to fewer accidents per car-mile but, unfortunately, far more catastrophic ones.
Re:Impossible to test (Score:5, Insightful)
Since when did that ever prevent anyone from doing anything? You must have us confused with some society that generally considers the full implications and long-term repercussions of our decisions...
Re:Impossible to test (Score:5, Insightful)
Your opinion of its likelihood is not relevant. Not only is it likely, evidence points to it being true. You are being disingenuous by phrasing it "no economic gains to be had by killing your customers." A product has a flaw, people die, that happens sometimes. If you issue a recall, you draw attention to the problem and cost yourself money in lost sales, repair costs, and possible lawsuits. "Killing your customers" is a bit different from "hoping that driver error is the official cause, not faulty cars," and you deciding to phrase it that way is an appeal to emotion, not a logical argument.
You can say we're just arguing semantics, but you're going to have to back up your unlikely opinion with links to convince me.
Re: (Score:3, Insightful)
The articles you linked indicate that Toyota has attempted a number of different fixes for what it believed were several separate acceleration related issues. I don't have to find links to make my case, you made it for me.
Re:Impossible to test (Score:5, Informative)
If a driver dials 911 on his cell phone before even trying to put the car in Neutral, then yeah - it is a driver error.
(The last case on the news - a driver called 911 on his cell phone because his car was accelerating out of control. When prompted by the operator if he had tried putting the car in Neutral, he said no and even refused to do so when ordered to do it by the operator.)
Re:Impossible to test (Score:4, Interesting)
You do realize that Prius's gear level is just a joystick, right? There is nothing mechanically connected, which means that if the computer is confused, then _there is nothing the driver can do_, except stomp on the breaks.
Actually there is. The car turns itself off if you hold the power button for 3 seconds. But in a panic situation, a person would most likely press the button repeatedly, instead of holding it steady for 3 long seconds. Other manufactures turn off on rapid button press in a short time, instead, which seems better.
Re:Impossible to test (Score:4, Insightful)
Judging by how the power button appears to work in the Prius, I would guess all that it does is tell the computer to shut down the engine. It's not like a typical car where the turning the key to Acc or off would cut the power to several critical systems. So if the computer is messed up to the point where the gear shift selector is not working, I wouldn't count on the power button to help you either.
Re:Impossible to test (Score:5, Interesting)
(The last case on the news - a driver called 911 on his cell phone because his car was accelerating out of control. When prompted by the operator if he had tried putting the car in Neutral, he said no and even refused to do so when ordered to do it by the operator.)
It's starting to look increasingly likely that this latest case was a hoax:
http://en.wikipedia.org/wiki/Toyota_Prius#Brake_fix_and_acceleration [wikipedia.org]
On March 8, 2010, a 2008 Prius allegedly uncontrollably accelerated to 94 miles per hour on a California Highway (US), and the Prius had to be stopped with the verbal assistance of the California Highway Patrol as news cameras watched [86]. Subsequent to the event, media investigations uncovered suspicious information about the alleged runaway Prius driver, 61-year old James Sikes, including false police reports, suspect insurance claims, theft and fraud allegations, television aspirations, and bankruptcy.[87][88] Sikes was found to be US$19,000 behind in his Prius car payments and had $US700,000 in accumulated debt.[87] Sikes stated he wanted a new car as compensation for the incident.[87][89] Analyses by Edmunds.com and Forbes found Sikes' acceleration claims and fears of shifting to neutral implausible, with Edmunds concluding that "in other words, this is BS",[90] and Forbes comparing it to the balloon boy hoax.[88]
Re:Impossible to test (Score:4, Informative)
You're one of those types that hit a wikipedia page, see some claimed fact without attribution, slap a [citation needed] on it, and then bugger off, aren't you?
You could just hit a search engine with some key words to see if you can find any corroborating source(s), of course:
http://news.google.com/news?q=toyota%20911%20neutral [google.com]
Oh hey, look at that.
- http://online.wsj.com/article/SB10001424052748704734304575120001542947616.html?mod=googlenews_wsj [wsj.com]
Re:Impossible to test (Score:5, Informative)
Every production car on the road has sufficient braking power to stall the engine in any gear at any throttle setting. Put your foot on the brake, and the car will stop. You may need new discs and pads after that.
Re:Impossible to test (Score:5, Interesting)
There are a couple of things that should be mentioned here. NASA has shown what it takes to make very small, very good code. Sure, they too have failures, but 'nearly' bug free code is quite expensive. Second, writing code is not quite like trying to create a hand crafted dashboard, if the dashboard fades, no one dies. Embedded software is quite a different beast from your normal desktop applications. When you add motion control and interaction with the code, it difference between them gets even more complex. Software in vehicles should be two things:
Open - let lots of folk see what could be wrong
Audited - audited to meet specific standards of safety and operation. Not quite the self-defeating government regulations, but more of a case by case issue: if the software has control or input to the control mechanism for the engine, braking system, suspension etc. it must meet minimum standard testing requirements. Any action that _could_ arbitrarily apply mechanical action must be tested and controlled beyond all reasonable testing/doubt. Everything should be tested, down to a pet chewing on the control cable harness.
Consumers are encouraged to think the vehicles they buy are safe and require no special knowledge of engineering or mechanics to operate. As long as they are given to think that, then passenger vehicles should be made to be just this way.
The problem for Toyota now is multifaceted. One, they have a PR shitstorm to deal with. Two, there is a dollar effect of this problem. Three, it's now on the shoulders of Toyota to get this part right for the rest of the passenger vehicle making industry.
It's possible that they might walk away from this fire with only minor long term burns and the reputation for building the safest vehicles. BUT, reading the article of this post and paying attention while doing so is necessary... IMO
Re: (Score:3, Interesting)
I'm still failing to see how the cars got locked in gear? every car i have driven has allowed the driver to shift the car into neutral regardless of everything else. This is both in automatics and definitely in my manual transmission cars(does anyone make a drive by wire clutch, outside of performance/race cars?) I fail to see why this is a huge issue that needs to be solved in the next 10 minutes and be 100%? how is a sticky peddle (software or otherwise) any different from the throttle body getting stuck
Re: (Score:3, Insightful)
The "PR shitstorm" is way way over-hyped, it would be simple for the news to simply state "Toyota has confirmed an issue effecting the engine speed controls, and have issued a recall. If this happens to you while driving Toyota advises drivers to shift the car into neutral and engage the 4 ways and pull over in a safe location. If your car has a push button start be aware that you will need to hold it down for up to 5 seconds to shut down the engine." The fact that some people have died as a result of poor
Re: (Score:3, Informative)
http://www.youtube.com/watch?v=NOMYjiCiTYg [youtube.com]
It doesn't, in fact, prevent you from shifting to neutral as you can tell by the sound of the engine butting heads with the rev-limiter.
Re: (Score:3, Insightful)
Actually, these days the engine won't blow if you suddenly shift into neutral due to rev limits in the engine controller, but that just makes any refusal to shift into neutral at speed even less excusable.
Defensive programming would include making a shift into neutral at speed override the accelerator position much like Toyota is doing (after the fact) with the brakes. Of course if the part of the software dealing with driver input is wedged, that won't help.
The other issue is the start button. Apparently y
Re:Impossible to test (Score:4, Informative)
Automatic trannies in these cars use the shifter as advice only. If shifted into neutral at speed, the engine would spin out, which is very damaging..
It doesn't seem you know much about cars. First of all, "spin out" isn't an automotive term. And 2nd, what do you mean by "spin out"? Modern cars have a rpm limiter which limits the RPM of the engine to some preset RPM limit. Have you ever driving a manual and hit the RPM limiter? It'll cut power to the engine. Same thing with neutral. An engine will not be damaged if you gas it in neutral, even to the RPM limit.
Re:Impossible to test (Score:5, Insightful)
Re:Impossible to test (Score:4, Funny)
``In exceptional conditions I would certainly include what the driver is wearing.''
I can see it now. Recall those drivers who called 911 when their vehicle accelerated out of control?
Driver: Help! My car is accelerating out of control!
911-person: What are you wearing?
Re: (Score:2)
Nothing is impossible. It's simply a matter of cost vs. benefit. Coding is many more lines today than it was 15 years ago. And a lot of copy paste of libraries you haven't coded or tested yourself. So ofcourse problems occur. But testing SHOULD be more than simply compiling the code.
Back in the day when I took my masters in computer science, we were forced to show how we tested EVERY loop, if, while or orhter "choice" or iteration of a possiblity. Simply to ensure that the code was 100% stable. Using closed
Re: (Score:2, Insightful)
Programmers today just give up too easily, and companies cut costs by scriping on testing, ducumentation and verification.
I'll start with the cheap joke - apparently you didn't learn the lesson too well.
And then the actual point - this is the real world, sir. Academics may have the luxury of taking as much time as they like to never produce result. That's not how things actually work for the rest of us.
Boeing versus Airbus (Score:5, Insightful)
I'm loving this conversation here because I've gotten crucified in slashdot before for making simmilar comments to the whole thread here. I grew up in a family of top managers of Boeing systems engineers. They hated computers. My dad never even learned how to turn one on. He hired other monkey to use the computers. As A child I was regailed with wonderful stories of every hard lesson in safety my dad had learned over his lifetime. He loved world war II because they got to use cutting edge designs for balls out performance yet at the same time learned how to make things reliable by disecting the accident. He would tell me about the accident that taught them that the engine pumps need to be at full speed but flow stalled on take off so that there's no lag when you hot swap after a pump fails. He told me of the accident where they learned not to route 100% of the control system wiring through any one junction box. etc...
Probably because of all these hard won lessons boeing for years insisted on fully mechanical or hydraulic flight surface controls. Whereas Airbus and other jumped on the fly-by-wire concept early. My dad would spit after hearing some youg person tout all the advantages of fly by wire. He knew them perfectly well. He was big on accepting new innovations to reduce fuel costs and increas performance. He was not a luddite. But he had a safety background that told him these electonic systems were hard as hell to validate and hard as hell to make truly independent from each other.
For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it. From what I've read it's now suspected that the latest airbus crash in the pacific had one of it's root problem in the voting nexus where a superior computer over ruled a more primitive safety system.
While we all know that computer software validation is hard if not impossible. It's not something we readily admit here on slash dot. It's because for years people like my dad would throttle the innovations the computer engineeers wanted to implement. I think as a result there became this culture of computer engineers that presented the case that embedded computing could be made safer than it really could be to offset that.
So now we come full circle and have to admit there is this middle ground. Just because a computer can improve perfromance does not mean it's reliable and safe. The old guys had a point after all when it came to safety.
Next week I'll tell you about how the ancient shocking lesson of the British Commet aluminum aircraft wings falling off led to the unanticipated discovery of metal fatigue and probably was the reason Boeing was slow to move to composite materials in commercial aircraft (but not in military aircraft). In hind sight we have heard of many tales of the composite tails of plane falling off as the reason for the loss of control before a crash. Conversely, composite wings on UAVs allow them to absorb a lot of bullet holes with no loss of control and to operate under higher perfromance conditions.
The point is that safety and performance are trade offs when both are pushed to the limit. The old guys know a lot more about safety than you might expect. The young guys are all about performance.
Re: (Score:2)
For example they often used triple redundant computers and if one of them disagreed the other two would vote it off the island and stop listening to it.
Sounds a little like Minority Report [wikipedia.org], doesn't it?
Each of the three precogs generates its own report or prediction. The reports of all the precogs are analyzed by a computer and, if these reports differ from one another, the computer identifies the two reports with the greatest overlap and produces a majority report, taking this as the accurate prediction of the future.
Re:Boeing versus Airbus (Score:5, Interesting)
A year ago I was watching one of Discovery programs I think and they had a couple of guys who supposedly implemented a piece of software, that would allow an airplane to fly and land safely if for some reason, while in the air, the tale would brake off or rudder would just stop working. They relied on a fly by wire airplane of-course and controlled the yaw with all other surfaces by applying very slight changes to the motion. They were saying a human could do this if extremely lucky, but software was able to do it almost always.
Just something to think about.
Re: (Score:3, Insightful)
Justifying something based on its performance on outlier conditions has never been a good idea. The number of accidents where the tail falls off is probably greater than the number of accidents where the tail falls off and the software would be able to compensate.
Maybe having the "oh shit" button turn this on would be a good idea, but I think if you look at the number of crashes and their causes, you'd want to build redundancy in the rudder or strengthen the tail.
Re: (Score:3, Interesting)
I have an example.
In a simulator (yes, a simulator) I was flying a VTOL type aircraft. Pulled a turn at too great a speed and broke off a few control surfaces. Maddening spin, completely unrecoverable (at least for me).
Tapped the button to enable "artificial stabilization" - which in this craft, enabled "puffers" charged with compressed air (driven by the engines, which still worked) - the computer control algorithms managed to use the remaining control surfaces and these puffers to level the craft and redu
Re:Boeing versus Airbus (Score:5, Insightful)
I'm pretty sure that the tail of an airplane falling off is an unanticipated scenario that humans cannot deal with either ;)
Re: (Score:3, Informative)
Yep, normally that results in death [wikipedia.org]
Re: (Score:3, Informative)
No, it was an example of one the computer CAN deal with (if it's anticipated in the programming.
A computer is limited by the creativity of the guy writing the IF THEN statements (setting aside the possibly of adaptive AI, but that raises other issues).
Other lessons from Boeing (Score:5, Interesting)
I worked on an embedded flight system there, and deeply respected people like your dad.
Boeing works under the eye of a certification authority who has to approve the safety of a design including, at least in the system I worked on, human factors. If there's anything comparable for cars, I haven't heard of it.
Boeing would not have made a pilot have to guess at how to turn an engine off (people with older cars, it's no longer a matter of turning a key).
Inputs were checked for consistency and validity. The specs would have anticipated what to do if the accelerator and brake were both full on at the same time.
There was a culture of worst-case planning and redundancy.
Also, if Boeing built a car, it would have a flight data recorder which investigators could examine and say for example "Looks like both(*) potentiometers on the accelerator went hard over at the same time, so we go look on the branches of the fault tree where there's a common-mode failure in the potentiometers or the pedal is down due to mechanical or pilot error".
(*) If I remember correctly from my obsessive pre-purchase research on Priuses, there are two separate sensors for accelerator position.
Re:Boeing versus Airbus (Score:5, Insightful)
Re: (Score:3, Insightful)
Amen. Fly by wire can never be perfectly safe - no matter how well a system is designed it can still fail. As long as its safer than the mechanical systems we're still ahead though.
Re: (Score:3, Insightful)
I agree with that wholeheartedly. Statements like "question all assumptions" are about as helpful as "foresee everything and never make mistakes."
I question that. It's just that the most complex tasks require software because they are beyond what can be done with mechanical syst
Re:Boeing versus Airbus (Score:5, Informative)
This one comment makes me wonder about the veracity of the balance of your account.
Then there is crunchy bit of FUD, which fails to mention that more than a few of those accidents are also associated with extreme control surface movements (inducing extreme stresses) prior to the failure.
Re: (Score:3, Informative)
Derek, as you might have noticed I was keeping it short on the comet disaster. But to expand. yes of course metal fatigue as a phenomena was known before the Comet disaster. What my Dad told me was they learned that they did not know about how to design for it yet. They did not have any computer modeling to know what flight stress really did to winds and to metal. they hardly had any way to measure material strength changes in-place. The people who built the Comet we no dummies so clearly they discove
Re:Boeing versus Airbus (Score:4, Insightful)
One can be short without being wrong. You were both short *and* wrong.
No, actually they discovered (as is widely documented in aviation histories) that they failed to correctly account for the stresses caused by multiple pressurization and depressurization cycles. They knew perfectly well how to design for metal fatigue, but lacked information on how that fatigue would manifest itself.
I merely pointed out how you have the story wrong, not that your point was false.
Re: (Score:2)
"Impossible to test", but that does not mean that it's impossible to write bug-free software. It requires a substantially different approach to specification and construction than most people/companies currently use. Model Checking (http://en.wikipedia.org/wiki/Model_checking) and SPARK (http://en.wikipedia.org/wiki/SPARK_(programming_language) ) are two approaches that work. It's worth looking at what the commercial avionics industry requires for its embedded software, where 10 ^ -9 is the requirement f
Re:Impossible to test (Score:5, Informative)
Indeed. I've done some embedded work myself. I wrote a power supply controller that used DACs to trim the voltage using some analog control ports on the DC to DC converter modules- it also monitored the PowerGood lines on the DC:DC's and linears and was programmed to shut down if one deasserted without a prior command telling it to do so. It had an I2C control network that could request status of bunch of aspects of the board including temperature, voltages, etc. Not wanting to risk blowing out a $10k FPGA with a $4.00 MCU, we had test boards with no FPGA on and some with cheaper FPGAs, and I also had a dev kit with the board on it hooked to a logic analyzer so we could emulate all sorts different scenarios and hopefully protect the FPGAs. Ultimately, a few problems emerged. With a particular combination of testing apparatus and polling rate, the I2C would receive interference and miss or corrupt some data. It was almost impossible to replicate reliably. This in turn exposed an oversight/bug where because of the skipped (as far as the power supply MCU was concerned) bytes, the wrong DAC values were being written, overvolting or undervolting the supplies- but it really only surfaced on the fully populated boards. This lead to a change in the I2C wiring/termination and a move to a keyed and transactional approach that required writing a key value to an address, writing the new data, then optionally reading back the data again, and lastly writing another key to a different address to either commit or roll back. Point is exactly what the parent said, it's very difficult to test some of these things because the problems may be an unusual chain of events or due to very specific circumstance in what's hooked to what and how much power is being drawn in the circuit at the time, etc.
The other portions of the code that performed monitoring and emergency shutdown caught the overvoltages very quickly and shutdown the FPGA in the span of a couple clocks. In the end we only lost one board, and it was due to ESD despite using proper handling techniques and equipment.
Re:Impossible to test (Score:5, Interesting)
Infallible fail. (Score:5, Insightful)
Re:Infallible fail. (Score:4, Informative)
i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...
Re:Infallible fail. (Score:4, Informative)
Re: (Score:3, Interesting)
i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, ...
Except we have testimony from any number of the Toyota acceleration victims that they had put the transmission into the "N" position, but the car just ignored it and kept accelerating. They also claimed that they knew how to use the brake, but the car also ignored that.
As a software guy, I'm quite familiar with ways that software will do things like this, and I find th
Re: (Score:2, Informative)
technically that is what part of the update does. It forces the computer to always choose the brake over the accelerator when both pedals are registering. So if the car does accelerate a tap on the brakes should disengage it.
Re:I agree on non-software fail-safes (Score:5, Insightful)
Re: (Score:2)
My VW doesn't care about the clutch being in OR the transmission being in neutral - you can use the starter to smash out the car in front of you in the parking lot.
Now the drive-by-wire? Any funkyness in input or measured output (throttle angle), and bam, I'm on the side of the road.
Re:I agree on non-software fail-safes (Score:4, Funny)
They should just implement a huge red panic button to shut everything off :)
They could even buy them at Staples [staples.com]. Need to stop your out of control Toyota? That's easy.
Another interesting statistic (Score:5, Interesting)
Some more data here [theatlantic.com]
Re:Another interesting statistic (Score:4, Insightful)
Be careful to note that the 24 cases discussed there are only the ones that have led to serious incidents.
Re: (Score:2)
Would that suggest a potential correlation between reaction time and general seriousness of the possible incidents?
Re:Another interesting statistic (Score:5, Informative)
To me it suggests that older drivers are having more difficulty coping with the situation once it arises.
Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:
http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes [forbes.com]
(They link the 911 recording:
http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/ [thetruthaboutcars.com]
)
So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.
But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).
Re:Another interesting statistic (Score:5, Insightful)
That part is strange. Uncontrolled acceleration is a much greater risk to life and limb than the red-lined/blown engine you might get if it were put into neutral with the throttle wide open. Being "afraid to try neutral" makes no sense.
Just an irrelevant side note: I've always found it low-class and tacky that phone calls made to 911 become publically available, especially when you hear them on the news. The message is, "hey sir or madam, remember that moment when you were highly emotional and had no idea if you were going to live or die? Well, we've got great news! That highly personal moment of reflection on your own mortality is now a public spectacle for millions of people! It's okay, we make a profit from this! No we won't share that profit with you..."
I realize it's a government service funded by taxpayer dollars. That explains how this is possible. It fails to explain how this is the best or most honorable thing to do.
That part generally shouldn't be a surprise. I'd imagine it also helps if you can keep calm and avoid panicking, as panicky people often fail at things they could do easily if they were not in a state of deer-in-headlights fear.
Nor does it explain why older drivers were disproportionately affected. Possibly the Toyota brand is more popular among older drivers because it historically has retained a decent resale value. While nothing the driver does should ever cause this kind of uncontrollable automatic acceleration, perhaps older drivers tend to have habits that somehow manifest whatever the actual underlying problem is. There are a lot of coincidences and correlations being pointed out in this discussion but unfortunately there seems to be little certainty about whether they are more than that.
Re:Another interesting statistic (Score:4, Funny)
Re: (Score:2, Informative)
Something to understand about those statistics: This is a self selected group based largely on income. Camrys may be everywhere but Prius' tend to be expensive.
Re: (Score:2)
Re: (Score:2)
2010 Camry base price: $19,595 - most expensive trim starts at $26,400
2010 Prius base price: $22,800 - most expensive trim starts at $28,070
Even a fully optioned Prius couldn't be described as expensive by new car standards. The price difference isn't enough to expect it would impact the type of owners very much... they fall in the same price range.
Testing. (Score:5, Insightful)
Logic of Testing (Score:5, Insightful)
David Cummings does seem to know what he's talking about, but as it is written, there is some strange logic in the article.
Testing cannot prove the absence of bugs, only their presence. There are two things that do not follow from this:
It sounds to me as if Toyota is saying the former, while Cummings says the latter. Neither is a correct conclusion.
Re:Logic of Testing (Score:5, Insightful)
Given practical software engineering conditions though, a) is highly unlikely while b) is highly likely.
Re:Logic of Testing (Score:4, Insightful)
I'll grant you that, but what I don't understand is this:
If you test, and do find some bugs, does that allow you to put any more trust in your software than if you tested and didn't find any?
Re: (Score:2)
That's a tricky question. Testing and finding bugs ought to allow you to put more trust in your testing methodology, which subsequently can increase your trust in your software once you stop finding bugs. Testing and finding no bugs, hard as you try, quite likely means you aren't trying hard enough. Very rarely will software reach a ceiling of reasonable test-proofness before being shipped that cannot be improved with subsequent, more dedicated, more specific testing after some issues are detected in the fi
Re: (Score:2)
Re: (Score:2)
So they speculate that code they wrote had an interrupt routine that was not bracketed with PUSHF/POPF instructions!!! Which is like Assembly 101.
I didn't read it that way. He's talking about an arithmetic carry condition being misremembered across an interrupt. This sounds to me like a CPU-internal hardware condition that might not have been included in the regular set of data that you save across an interrupt. Maybe the wrong type of PUSHF/POPF was used. It certainly doesn't sound like Assembly 101.
Help me benefit from media hype (Score:2)
I'm in the market for a car and everyone is picking on Toyota now.
I don't believe in stupid media hype. I don't believe cars rewire themselves. And I know how to hit the breaks, shift into neutral, and/or turn off the key when I want the car to go slower (so far, hitting the breaks has always proven adequate).
Are there any really good deals on Toyotas available?
Re: (Score:2)
Be careful about switching off the ignition while you're still moving. It could cause loss of steering and/or braking ability. That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.
Re: (Score:2)
Not quite true. You may lose "power assisted" braking and steering but the wheel will still steer the car and the brakes will still work, it will just take a little more effort. Those old enough will remember a time before power assisted steering and braking.
Re: (Score:2)
Re: (Score:3, Informative)
Any vehicle built in the last thirty or fourty years will not allow the steering column to lock unless the transmission is in park. If you're in drive (or neutral) you can only turn it to "off", not all the way to "lock". This was to prevent an errant knee from locking the steering while you're doing 70 on the freeway. Happened to me once, except I was only doing 45 on a bumpy ass gravel road when my knee smacked into my keychain. It was startling, but not particularly dangerous.
Re: (Score:2)
Yeah, it has never actually been necessary to turn off the ignition. But my engine has killed while moving before. Steering and brakes still function, not as well, but good enough for a safe stop.
Software failed to fix my spelling of "brakes"! Someone call David Cummings!
Re: (Score:2)
That's b-r-a-k-i-n-g, not b-r-e-a-k-i-n-g.
Lack of one leads to the other. I'm sure he'll figure it out ... I hope.
Since the actual problem hasn't been identified yet, who knows if all Toyotas have it or just a select few? Maybe it's a date/time bug? Integer overflow bug? I don't know what the problem is, but I know where it is and I'm not willing to put my family at risk in order to secure a deal on a car that may be more dangerous than most.
Re: (Score:3, Interesting)
I used to have a car where the engine would suddenly turn off for no reason while driving, often at exciting moments like getting onto the freeway. It was pretty easy to put it into neutral (it was an automatic), turn the key to "acc" and try to restart the engine (usually with success) without accidentally locking the steering wheel.
It went on for some time until I convinced the repair guys to clean all the electrical connections from the computer to the fuel pump. The car had lived most of it's life in
Re:Help me benefit from media hype (Score:4, Informative)
> And I know how to hit the brakes...
With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine (this is, IMHO, a fundamental design defect).
> ...shift into neutral...
The computer may not let you do that with the car moving and the engine at high rpm. After all, the engine and/or transmission might be damaged (another design defect).
> ...and/or turn off the key...
Some of these vehicles don't have keys: just a radio remote. The emergency shutdown procedure is to hold a button down for three seconds (another design defect).
Re:Help me benefit from media hype (Score:4, Informative)
> With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine
apparently not true [popularmechanics.com]
Obama's solution (Score:2)
It seems he wants more software. This time to check for both pedals being depressed at the same time. One more thing to break.
The less these embedded systems have to do, the better.
Smaller Systems Solution? (Score:2)
Didn't we already "solve" this problem in the airline industry by breaking down larger more complex software into separate physical components that can be more easily verified? Can we do the same with cars?
Just split one computer into half a dozen much smaller, simpler, little units and set the valid IO conditions for them, then have the components around them sanity check their output.
Are the brakes totally drive-by-wire as well? (Score:2)
Pardon my laziness for not investigating this myself, but doesn't the Prius have a mechanical (hydraulic) link to the brakes that engages when the pedal is pushed down far enough? I realize that the first portion of the braking is done electronically (for the regenerative braking system), but in an emergency wouldn't a full application of the brakes slow down the vehicle?
From the days of "winmodems" (Score:2, Interesting)
I've said time and time again, "Never replace hardware with software" because
something dedicated to the task will always work better, or be less failure
prone (more often than not).
Would Toyota be having these problems with an accelerator cable vs electronic?
99% sure the answer is "no"...heck the solution is add some grease, make sure
it isn't pinched/looped too tightly and/or add tension to the pedal side.
Or, replace the damn cable with a new one...a 20 to 30 minute task.
(less than 10min on a motorcycle)
Oh,
Re:From the days of "winmodems" (Score:4, Insightful)
On the flip side, the USN replaced complicated and heavy hardware analog computing systems for [SSBN] missile guidance systems with software running on a digital computer, and MTBF went through the roof and maintenance man hours and MTTR through the floor. The same thing happened when they replaced the analog torpedo fire controls with digital ones. The same thing happened again when the hovering system controls were upgraded to digital.
Now, before you claim that is a limited set of examples, I invite you to consider the millions of incident free flight hours accumulated by fly-by-wire aircraft. Or the replacement of DIP switches in PC's with software configuration. Etc... Etc...
Re: (Score:3, Interesting)
> Would Toyota be having these problems with an accelerator cable vs
> electronic?
GM once had a very similar problem with a 70s car with a cable. An engine mount failure would allow the engine to rotate under acceleration in such a way as to yank the cable to full throttle and then jam it, causing the car to run away. The resulting collision would knock the cable free and as collisions often break engine mounts, the evidence disappeared.
Computerized systems are usually more reliable than mechanical o
God damn legal system (Score:2, Insightful)
Metric unit (Score:2)
prove a negative? (Score:3, Insightful)
Isn't this like proving God doesn't exist?
They can test and test and not get a result that said this is the bug, so they assume that it doesn't exist.
I wonder if (Score:2)
Speaking as an embedded programmer here... (Score:4, Interesting)
... why does everyone assume it is a software bug? I agree that it very well could be an undiscovered software bug. But there are so many more sources of erroneous behavior in an embedded system that *even* *if* the software were flawless (ummm... just go with me a minute... :) an automotive environment can cause all manner of strange glitches. I work with robots, lots of DC motors causing commutation noise on the power supply, long (several inch) distances between units that must talk to each other and therefore may have a different opinion as to ground reference voltage... many things can get wacky. Even flawless code needs a watchdog timer to get you out of weird states that power glitches that put you into. Power supply spikes can cause the program counter to jump to very odd places, with odd, corrupted stuff in RAM. Ground level shifting can cause communication glitches. CAN bus is *extremely* robust, so bad data should not get through... but what does get through? Does the system as a whole get into a weird state if packets drop?
Story of an elusive bug... (Score:3, Informative)
Several years ago, I designed the software for a real-time automotive test system called HP ECUTEST (I think the official name was HP Design Span DS5470, but let's not waste time on HP's cold dead fish naming conventions). It simulated a car from an electric point of view. You connected an electronic control unit (ECU), and it had basically no way to tell it was not in a real car. Think of it as The Matrix for car electronics.
One of our first customers wanted us to test it with a reliable, proven, tested, tried and true ECU, something that was on the road in cars for several years already. So we did. And I noticed something odd. The ECU worked fine when we "drove" a car normally, but at idle, it would basically slow down, one RPM at a time, until it stopped. However, if I changed the value of the input corresponding to the accelerator pedal, it would reset the idle speed to the default, something like 800rpm.
Finally, after eliminating the possible bugs on our side, we tell the customer. Their first reaction was "no way". But after a week and a demo of the problem, they finally made a connection. They had this elusive bug of some car customers complaining that their car would sometimes stop when idle. It turns out that in a real car, chassis vibrations generally caused minute changes in the input value for the accelerator. So the ECU would correctly recompute its idle speed. However, if there was no change, like if the pedal was more rigid than usual, the bug would trigger.
The root cause was a routine that wanted to optimize idle speed to be as low as possible, but for some reason kept cached data if the accelerator had not changed, so it thought the engine was still running smoothly.
We found such bugs in practically all ECUs we tested for the first time. The most impressive one was in a V8 ECU that was basically a V8 until 1200rpm, then a V7, then a V6, and basically a V2 above 4000 rpm. The customer had hoped we'd find something, because they didn't get all the power they expected from the engine. Obviously. It was hard to find without our system, because the injectors that fired were differnt from cycle to cycle, so more simple instrumentation saw all cylinders running. The root cause here was that the software badly exceeded its real-time envelope... Ouch.
It is possible, and is done, but hard/expensive (Score:3, Interesting)
I do industrial automation for a living, and machine guarding/safety is a major component of the job. There are now, in the last few years, software based safety products that are provably just as safe as a hardware only safety products. The key is that it's not just about rigorous testing, it's about correct design. If you want category 4 protection, you need to be sure that:
Software becomes another component. Therefore you need to have redundancy in your software. Government regulators that certify these safety systems as compliant want to see you prove that a single component (i.e. unit of software) can't malfunction and leave the system in an unsafe state. What a lot of companies do is they have two independent processors each monitoring the inputs to the system in parallel, and each generating the required outputs. The processors are typically sourced from different companies, and the circuit boards are designed by different teams. The software running on each processor is written by a different team. If both processors agree on the outputs, the system drives those outputs, and if not, all power is dropped to everything and the system can't be restarted (may need to be replaced, etc.).
Those of us in the industry were skeptical of software based safety at first, but given the above facts and a decent amount of regulatory oversight, I'm satisfied that it will live up to the design criteria. That doesn't mean an error can't happen, but it makes the probability low enough that we can live with it.
The latest thing is safety systems running their I/O across networks like DeviceNet and even Ethernet/IP (the IP stands for Industrial Protocol, not Internet Protocol). Again, I was at first skeptical, but they use a protocol layering on top of the network using timestamps and redundant processors on both ends with reasonable failure modes that the system is provably safe, within reasonable limits.
So you can make safe embedded systems, but without being able to inspect the design and see that it lives up to these guidelines, Toyota can't ever *prove* that the system is safe.
A Most Unusual Bug Indeed (Score:3, Interesting)
The Times also helpfully provides a list of all the people who have died in "sudden acceleration" accidents involving Toyotas:
Toyotas, deaths and sudden acceleration [latimes.com]
If you look through the list at the ages mentioned, one begins to notice a rather odd pattern: 18, 21, 32, 34, 44, 45, 47, 56, 57, 58, 60, 61, 63, 66, 68, 72, 72, 77, 79, 83, 85, 89
This is a most peculiar bug indeed in that it seems occur primarily when the driver is elderly. Or perhaps, as with previous "sudden acceleration" scares, this will ultimately turn out to be the result of people slamming on the gas when they menat to slam on the brake and then trying to blame the car for their error.
pay for QA and don't over work coders that makes b (Score:2)
pay for QA and don't over work coders that makes bugs. When people are working 80+ hours they make more bugs then people doing 40. Also pay for QA and don't cut back there. Look at the xbox 360 to see what that left them with.
Re: (Score:2, Insightful)
The type of people that purposely hide bugs that will likely kill several people are the same type of people you can't really "appease" no matter what you do.
Re: (Score:2)
you want just plain silly the computer models they use are different from the ones in use by the weather dept. The weather Dept is only good for about 3 days ahead. If our current setups are good for just days in advance why do they think that their models are accurate over thousands of years?