Toyota Acceleration and Embedded System Bugs 499
An anonymous reader writes "David Cummings, a programmer who worked on the Mars Pathfinder project, has written an interesting editorial in the L.A. Times encouraging Toyota to drop claims of software infallibility in their recent acceleration problems. He argues that embedded systems developers must program more defensively, and that companies should stop relying on software for safety. Quoting: 'If Toyota has indeed tested its software as thoroughly as it says without finding any bugs, my response is simple: Keep trying. Find new ways to instrument the software, and come up with more creative tests. The odds are that there are still bugs in the code, which may or may not be related to unintended acceleration. Until these bugs are identified, how can you be certain they are not related to sudden acceleration?'"
Impossible to test (Score:5, Informative)
Plus, all this hype around these Toyota acceleration problems is just that, hype.
Re:Impossible to test (Score:5, Informative)
Re:I agree on non-software fail-safes (Score:2, Informative)
technically that is what part of the update does. It forces the computer to always choose the brake over the accelerator when both pedals are registering. So if the car does accelerate a tap on the brakes should disengage it.
Re:Another interesting statistic (Score:2, Informative)
Something to understand about those statistics: This is a self selected group based largely on income. Camrys may be everywhere but Prius' tend to be expensive.
Re:Infallible fail. (Score:4, Informative)
i'd feel much better with drivers who know they should pop the car into NEUTRAL if it starts accelerating out of control for any reason, rather than trying to stand on the brake pedals while dialing 911 ...
Re:Another interesting statistic (Score:5, Informative)
To me it suggests that older drivers are having more difficulty coping with the situation once it arises.
Forbes says that the guy who got himself plastered all over cable last week was 'afraid' to put the vehicle into neutral, or to turn off the engine:
http://www.forbes.com/2010/03/12/toyota-autos-hoax-media-opinions-contributors-michael-fumento.html?boxes=financechannelforbes [forbes.com]
(They link the 911 recording:
http://www.thetruthaboutcars.com/the-jim-sikes-911-call-23-minutes-of-unintended-acceleration/ [thetruthaboutcars.com]
)
So apparently being an idiot is also a likely factor in the failing to cope with the incident before it becomes lethal.
But they key observation is that the higher number of fatalities among older drivers doesn't really point to the source of the problem being driver error (rather, the driver error is in failing to deal with the situation once it arises).
Re:Help me benefit from media hype (Score:4, Informative)
> And I know how to hit the brakes...
With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine (this is, IMHO, a fundamental design defect).
> ...shift into neutral...
The computer may not let you do that with the car moving and the engine at high rpm. After all, the engine and/or transmission might be damaged (another design defect).
> ...and/or turn off the key...
Some of these vehicles don't have keys: just a radio remote. The emergency shutdown procedure is to hold a button down for three seconds (another design defect).
Re:Impossible to test (Score:5, Informative)
Indeed. I've done some embedded work myself. I wrote a power supply controller that used DACs to trim the voltage using some analog control ports on the DC to DC converter modules- it also monitored the PowerGood lines on the DC:DC's and linears and was programmed to shut down if one deasserted without a prior command telling it to do so. It had an I2C control network that could request status of bunch of aspects of the board including temperature, voltages, etc. Not wanting to risk blowing out a $10k FPGA with a $4.00 MCU, we had test boards with no FPGA on and some with cheaper FPGAs, and I also had a dev kit with the board on it hooked to a logic analyzer so we could emulate all sorts different scenarios and hopefully protect the FPGAs. Ultimately, a few problems emerged. With a particular combination of testing apparatus and polling rate, the I2C would receive interference and miss or corrupt some data. It was almost impossible to replicate reliably. This in turn exposed an oversight/bug where because of the skipped (as far as the power supply MCU was concerned) bytes, the wrong DAC values were being written, overvolting or undervolting the supplies- but it really only surfaced on the fully populated boards. This lead to a change in the I2C wiring/termination and a move to a keyed and transactional approach that required writing a key value to an address, writing the new data, then optionally reading back the data again, and lastly writing another key to a different address to either commit or roll back. Point is exactly what the parent said, it's very difficult to test some of these things because the problems may be an unusual chain of events or due to very specific circumstance in what's hooked to what and how much power is being drawn in the circuit at the time, etc.
The other portions of the code that performed monitoring and emergency shutdown caught the overvoltages very quickly and shutdown the FPGA in the span of a couple clocks. In the end we only lost one board, and it was due to ESD despite using proper handling techniques and equipment.
Re:Help me benefit from media hype (Score:3, Informative)
Any vehicle built in the last thirty or fourty years will not allow the steering column to lock unless the transmission is in park. If you're in drive (or neutral) you can only turn it to "off", not all the way to "lock". This was to prevent an errant knee from locking the steering while you're doing 70 on the freeway. Happened to me once, except I was only doing 45 on a bumpy ass gravel road when my knee smacked into my keychain. It was startling, but not particularly dangerous.
Re:Help me benefit from media hype (Score:4, Informative)
> With the engine past the redline there is very little vacuum to operate the power brakes. Without power assist the brakes may not be able to overcome the engine
apparently not true [popularmechanics.com]
Re:Boeing versus Airbus (Score:3, Informative)
Yep, normally that results in death [wikipedia.org]
Re:Boeing versus Airbus (Score:5, Informative)
This one comment makes me wonder about the veracity of the balance of your account.
Then there is crunchy bit of FUD, which fails to mention that more than a few of those accidents are also associated with extreme control surface movements (inducing extreme stresses) prior to the failure.
Re:Infallible fail. (Score:4, Informative)
Re:Impossible to test (Score:2, Informative)
Automatic trannies in these cars use the shifter as advice only. If shifted into neutral at speed, the engine would spin out, which is very damaging. So the software prevents that. Oops. Note, most cars in the US have automatic transmissions.
Still, the best steps are check the pedal, try neutral, try cutting the engine (tap tap tap, hold, or whatever it takes if it is a button), and use the brake forcefully and completely, before the pads have a chance to heat up. Assuming you have the /!right/ pedal.
Re:Toyota: (Score:3, Informative)
And more examples of how wrong things can get can be found here: http://thedailywtf.com/ [thedailywtf.com]
There are some good examples there, but you'll find more on comp.risks [ncl.ac.uk].
Story of an elusive bug... (Score:3, Informative)
Several years ago, I designed the software for a real-time automotive test system called HP ECUTEST (I think the official name was HP Design Span DS5470, but let's not waste time on HP's cold dead fish naming conventions). It simulated a car from an electric point of view. You connected an electronic control unit (ECU), and it had basically no way to tell it was not in a real car. Think of it as The Matrix for car electronics.
One of our first customers wanted us to test it with a reliable, proven, tested, tried and true ECU, something that was on the road in cars for several years already. So we did. And I noticed something odd. The ECU worked fine when we "drove" a car normally, but at idle, it would basically slow down, one RPM at a time, until it stopped. However, if I changed the value of the input corresponding to the accelerator pedal, it would reset the idle speed to the default, something like 800rpm.
Finally, after eliminating the possible bugs on our side, we tell the customer. Their first reaction was "no way". But after a week and a demo of the problem, they finally made a connection. They had this elusive bug of some car customers complaining that their car would sometimes stop when idle. It turns out that in a real car, chassis vibrations generally caused minute changes in the input value for the accelerator. So the ECU would correctly recompute its idle speed. However, if there was no change, like if the pedal was more rigid than usual, the bug would trigger.
The root cause was a routine that wanted to optimize idle speed to be as low as possible, but for some reason kept cached data if the accelerator had not changed, so it thought the engine was still running smoothly.
We found such bugs in practically all ECUs we tested for the first time. The most impressive one was in a V8 ECU that was basically a V8 until 1200rpm, then a V7, then a V6, and basically a V2 above 4000 rpm. The customer had hoped we'd find something, because they didn't get all the power they expected from the engine. Obviously. It was hard to find without our system, because the injectors that fired were differnt from cycle to cycle, so more simple instrumentation saw all cylinders running. The root cause here was that the software badly exceeded its real-time envelope... Ouch.
Re:Impossible to test (Score:5, Informative)
If a driver dials 911 on his cell phone before even trying to put the car in Neutral, then yeah - it is a driver error.
(The last case on the news - a driver called 911 on his cell phone because his car was accelerating out of control. When prompted by the operator if he had tried putting the car in Neutral, he said no and even refused to do so when ordered to do it by the operator.)
Re:Boeing versus Airbus (Score:3, Informative)
No, it was an example of one the computer CAN deal with (if it's anticipated in the programming.
A computer is limited by the creativity of the guy writing the IF THEN statements (setting aside the possibly of adaptive AI, but that raises other issues).
Re:Boeing versus Airbus (Score:3, Informative)
Derek, as you might have noticed I was keeping it short on the comet disaster. But to expand. yes of course metal fatigue as a phenomena was known before the Comet disaster. What my Dad told me was they learned that they did not know about how to design for it yet. They did not have any computer modeling to know what flight stress really did to winds and to metal. they hardly had any way to measure material strength changes in-place. The people who built the Comet we no dummies so clearly they discovered a scaling issue in metal no one had encountered before nor new how to design for at that time. The point I was making was that just because you can't foresee a problem in something new does not mean you cant anticipate there might be problems you can't foresee. The switch to composites opened up the same sort of issues that the comet did.
Re:Impossible to test (Score:5, Informative)
Every production car on the road has sufficient braking power to stall the engine in any gear at any throttle setting. Put your foot on the brake, and the car will stop. You may need new discs and pads after that.
Re:Impossible to test (Score:3, Informative)
http://www.youtube.com/watch?v=NOMYjiCiTYg [youtube.com]
It doesn't, in fact, prevent you from shifting to neutral as you can tell by the sound of the engine butting heads with the rev-limiter.
Re:Other lessons from Boeing (Score:2, Informative)
Also, if Boeing built a car, it would have a flight data recorder which investigators could examine and say for example "Looks like both(*) potentiometers on the accelerator went hard over at the same time, so we go look on the branches of the fault tree where there's a common-mode failure in the potentiometers or the pedal is down due to mechanical or pilot error".
From working in the field of emergency response, quite a number of newer cars (somewhere around 2000 they started putting them in) do have a "blackbox" of some kind (some more detailed than others). Having said that, I'm not the one to examine them nor do I pull them out of cars after an accident. But, having talked with the guys that do, they get a surprising amount of data from them. It tells them whether or not the air-bags were deployed, the highest speed before "sudden" deceleration, the time it took the car to come to rest, whether the breaks had been applied, whether the seatbelts were engaged, traction of the tires (if the car has one of those "smart" traction-control features), and all time-stamped (sure there is more, but that's all I can remember from the conversation).
Just thought I would point that out for those that are unaware.
Cheers.
Re:Impossible to test (Score:1, Informative)
bzzzt.... wrong. What happens to your vacuum assisted brakes when the throttle is wide open? I'll give you a hint... they are no longer vacuum assisted. At that point, however hard you push on the pedal is how hard the brakes are applied. Perhaps your assertion is true if the car is starting at rest (where the brakes would be combating off-idle torque) but, at speed, a manual set of brakes would have difficulty overcoming 600 lb-ft of torque (200 engine torque X 3:1 final drive) plus vehicle inertia. The pads would quickly overheat and out-gas reducing braking force. Continued application of brakes would then cause the fluid to boil causing complete brake failure.
Re:Impossible to test (Score:4, Informative)
Automatic trannies in these cars use the shifter as advice only. If shifted into neutral at speed, the engine would spin out, which is very damaging..
It doesn't seem you know much about cars. First of all, "spin out" isn't an automotive term. And 2nd, what do you mean by "spin out"? Modern cars have a rpm limiter which limits the RPM of the engine to some preset RPM limit. Have you ever driving a manual and hit the RPM limiter? It'll cut power to the engine. Same thing with neutral. An engine will not be damaged if you gas it in neutral, even to the RPM limit.
Re:Impossible to test (Score:4, Informative)
You're one of those types that hit a wikipedia page, see some claimed fact without attribution, slap a [citation needed] on it, and then bugger off, aren't you?
You could just hit a search engine with some key words to see if you can find any corroborating source(s), of course:
http://news.google.com/news?q=toyota%20911%20neutral [google.com]
Oh hey, look at that.
- http://online.wsj.com/article/SB10001424052748704734304575120001542947616.html?mod=googlenews_wsj [wsj.com]
Re:Impossible to test (Score:3, Informative)
Except that drive-by-wire systems have absolutely no possibility of making the operation of the car safer (more efficient, sure, but that's it) and many many ways in which to make it less safe, so you're wrong about it making more sense to use drive-by-wire. I can't figure out what you're thinking with that statement.
That's flat-out not true. Traction Control and brakes and power synchronizing together (which is a lot of what drive-by-wire DOES) absolutely, in most cases, makes the average driver safer. Personally, I'm inclined to think that as drive-by-wire improves (until eventually, hopefully, the really dangerous part (i.e. the driver) will go away), we will in general progress to fewer accidents per car-mile but, unfortunately, far more catastrophic ones.