Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Programming IT

How Did Volkswagen Cheat Emissions Tests, and Who Authorized It? 618

Lucas123 writes: The method by which Volkswagen diesel cars were able to thwart emissions tests and spew up to 40X the nitrogen oxide levels set by the Environmental Protection Agency was relatively simple. It was more likely no more than a single line of code used to detect when an emissions test was being performed and place the emissions system in an alternate mode — something as simple as a software "on/off" switch. Volkswagen AG CEO Martin Winterkorn, who stepping down as the result of his company's scandal, has said he had no knowledge of the emissions cheat, but software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it. You can actually see who asked the developer to write that code," said Nikhil Kaul, a product manager at test/dev software maker SmartBear Software. "Then if you go upstream you can see who that person's boss was...and see if testing happened...and, if testing didn't happen. So you can go from the bottom up to nail everyone."
This discussion has been archived. No new comments can be posted.

How Did Volkswagen Cheat Emissions Tests, and Who Authorized It?

Comments Filter:
  • Nail everyone? (Score:5, Insightful)

    by Anonymous Coward on Thursday September 24, 2015 @09:28AM (#50589183)

    Correction: "You can nail everyone that's in the official audit trail."

    The people at the top that authorized it (or at least didn't condemn it) probably never actually sent a traceable e-mail to anyone. Nor did they touch any code. Nor do they appear in any meeting minutes. These sorts of discussions tend to happen over a drink in a bar somewhere, and for good reason.

    • seriously, if you're a dev and PHB asks you to do this then you make sure you get it in email form and then forward that email to a personal account or to a safe pst file you keep yourself. or you send an email to said PHB saying what you did. along with everyone on the testing and QA team. there was a paralegal arrested in NYC recently who forged a judge's signature without his bosses knowing because he was swamped with work and in his mind it was the only way to do it knowing he would be cause and commit
      • Re:Nail everyone? (Score:5, Interesting)

        by tripleevenfall ( 1990004 ) on Thursday September 24, 2015 @09:40AM (#50589299)

        I agree, but in real life things are often not this simple. You get requirements that you aren't sure are a good idea or the right thing to do, and you question them - but PHB assures you that it's all been approved and cleared by the people on higher floors, and you may even contact some of them and hear their agreement. You're a coder and now a lawyer, so...? You go ahead and write IF OBDIIPortHasSomethingConnectedToIt THEN EnterDiagnosticMode .

        This case seems very egregious, but the truth of ethics in real life is often difficult to determine, and it's being thought about by a human whose livelihood may depend on the choice.

        It's not hard to see how things like this happen. People will almost always act in (what they think is) their own best interest.

        • as long as you have this in writing and can give it to a lawyer that is the most important thing. otherwise PHB will leave you out to dry saying they gave you no such order or instruction
          • Re:Nail everyone? (Score:5, Informative)

            by pla ( 258480 ) on Thursday September 24, 2015 @10:53AM (#50589877) Journal
            The hard part here comes from "get it in writing".

            When someone three layers of food-chain above you tells you "do this", you don't get to refuse until you have it in writing (unless you already have a new job lined up - and even then, don't expect that one to go any differently).

            Now, you can certainly try to get them on record - You can ask them to write up a quick spec for what they want; you can ask them to submit the Change Management request because you don't have the authority to approve this one; you can send emails asking for clarification; and as a last resort, you can just document the change as "at the request of Boss X". In the real world, however, we've all dealt with people who refuse to do anything except by phone or in person.

            And at that point, it becomes your word against theirs. Guess who can afford the better lawyer? And even that assumes it completely blows up - If it remains an internal matter, you won't even get the chance to present your side of the situation, just pack your belongings up and GTFO.
            • by Joe_Dragon ( 2206452 ) on Thursday September 24, 2015 @11:17AM (#50590101)

              Professional Engineers have the power to say no and they have Ethics rules to fall back on.

              • by BasilBrush ( 643681 ) on Thursday September 24, 2015 @11:56AM (#50590387)

                I find it odd to hear how programmers seem so abused by PHBs. Maybe it's an American thing, but in the UK, I've always found that employers want to keep hold of skilled people like programmers, because new ones are hard to find and take a while to get up to speed. This means that saying no is always possible.

                (Nothing to to with official engineer status and ethics. There's no general requirement for engineer certifications for programmers here.)

                • by Grishnakh ( 216268 ) on Thursday September 24, 2015 @03:53PM (#50592111)

                  It's an American thing.

                  Employers here don't care about holding onto skilled programmers or other skilled people, because PHBs think they can just hire replacements on a whim.

                  Yes, in reality new ones are hard to find and take a while to get up to speed. The PHBs will even acknowledge this when they're trying to hire.

                  But once they have one employed, they don't care about keeping him happy, because they think they're al interchangeable cogs.

                  If you're seeing a giant disconnect here, yes, there is. This is how American corporations think; it makes no sense at all. I can't explain it. It's the same phenomenon where corporations will give a big salary offer to a new engineer, but once he's employed there, they'll just freeze his salary or give him paltry CoL raises, while giving new hires even bigger salaries, causing employees to switch jobs every 2-4 years (in Silicon Valley, it's 12-18 months).

              • by afeeney ( 719690 ) on Thursday September 24, 2015 @12:39PM (#50590685)

                It's fascinating to see how many posters here automatically assume that it must be the PHBs who pressured the engineers into this. Very few assume that the engineers saw an opportunity for a bonus or for the PHB to owe them one, and added the cheat function voluntarily. I've not seen any posts so far that suggest an engineer thought of the cheat and suggested it to a PHB.

                A reminder that we tend to think of our peers as being much more ethical than "them" and look for reasons to think of them as victims of force or circumstances, and assume that "they" are only motivated by sheer callous greed. Whoever the "them" is.

                • by AK Marc ( 707885 ) on Thursday September 24, 2015 @01:46PM (#50591155)
                  And I find it hilarious that everyone here states that programmers and engineers work without requirements or documentation. I've worked places where the verbal meeting would have the engineer agreeing with everything, then when it's not written in the requirements document and signed off by 10+ people, it doesn't get built. Seems like all the programmers on Slashdot have never worked in a company larger than 10 people.
              • by NoKaOi ( 1415755 )

                Professional Engineers have the power to say no and they have Ethics rules to fall back on.

                But do they have the power to say no and keep their job, and keep their job without management making their work life miserable?

              • by goombah99 ( 560566 ) on Thursday September 24, 2015 @04:39PM (#50592471)

                While it is likely this was a sin of commission it remains plausible that no one did this at all. My thinking is that if instead of being programmed explicity the computer program was allowed to train itself for it's emission and performance tuning that a very natural outcome would be for it to learn to minimize emissions during emission type testing. Then on the test track it would learn performance and handling. etc... and so you end up with something that cheats but no one told it to nor was anyone even trying to make it cheat. It's just the result of getting what you optimize for.

                One reason that I like that theory is that if you consider the opposite, that it was a conspiracy, then this is not the sort of thing you can keep secret easily. You might succeed but that's pretty hard especially considering the time span and the inevitable entry of new personnel and suppliers into the supply chain. So I don't think this was intentional. The exception might be if if it's a conspiracy of one. that for some reason there was just one guy who could pull off everything. THen you would have a shot of keeping this secret.

            • Re:Nail everyone? (Score:4, Interesting)

              by kaatochacha ( 651922 ) on Thursday September 24, 2015 @11:54AM (#50590369)

              Then you do the next best. You get it in writing from your immediate supervisor, playing as if you're verifying facts of the request. Someone that is used to receiving such requests from you. He/she, in turn, realizes what you are doing, and does the same with His/her immediate supervisor. This continues up the chain until it eventually reaches the upper management who requested it. If someone along the line doesn't do this, they become the scapegoat, not you.
              It's a complete CYA action.

            • As an American who has worked for two different large corporations, I am surprised by this sentiment. I once got an order to do something I felt was unethical from my dotted-line boss. I sent a note to my boss immediately and his response was, literally, "If you do that, I will fire you." There was no pressure to do anything that even remotely seemed like a Bad Thing (TM).

              Now that I'm a little higher up the food chain, I get the occasional request for my team to do stuff that's not in keeping with ethical o

            • The hard part here comes from "get it in writing".

              Not really. What I've done in very "interesting" situations (names will be withheld), is that I send an e-mail that would have the following structure (I'm putting the important part in bold):

              As per our discussion/instructions today, we will implement X,Y,Z and that this has been cleared/authorized/whatever-adjective-you-see-fit, and that these steps are appropriate. (If there are any lingering concerns, I list them here.)

              Please let me know if you have any questions or corrections. Otherwise, I will proceed with your approval.

              Once I send that e-mail, I reply to it, to me, indicating the time and date the discussion/instruction took place.

              The critical part is in bold. No reply to the contrary implies tacit approval. Not bullet proof, and I'm no lawyer, but I've gotten people to backtrack "strange" orders as soon as they get such an e-mail.

          • Re:Nail everyone? (Score:5, Insightful)

            by Eunuchswear ( 210685 ) on Thursday September 24, 2015 @11:23AM (#50590151) Journal

            Oh, great. If you have it in writing you can say "I was just obeying orders". In a fucking German accent.

            That'll go down well.

            • Re:Nail everyone? (Score:5, Insightful)

              by orgelspieler ( 865795 ) <w0lfieNO@SPAMmac.com> on Thursday September 24, 2015 @12:30PM (#50590637) Journal

              That's ridiculous. There are valid reasons for the code to have different operating modes. It is not inherently unethical to make software behave differently under different operating conditions.

              Think about it, VW sells cars in multiple jurisdictions. They need to have different emission mitigation regimes for each of them. It's possible that the flags for the operating conditions just got screwed up. If they use the same shop/lab model to pass EU and USA testing, there must be a switch in there for "Detected_testing_regime == EPA." And they might have intended to turn on the same controls when "vehicle_sold_to == USA," but the guy responsible for that code screwed something up.

              Even if it were an intentional deception, it would be trivial for management to divvy up the workload such that the individual programmers had no idea. "Jan, we need you to make a module that detects the following operating conditions and pass this parameter to the emissions controller, mm-kay?" or "Hans, when you see the disable_emission_control_for_emergency_use, make sure that the car switches to the high_power curve."

              We really don't know. Until the results of the investigation are made public, and they are found guilty of wrong-doing, they are innocent. Is the legal equivalent of "assume good faith."

              • by NoKaOi ( 1415755 )

                And even then, a programmer or his/her manager could claim that code was for testing purposes, it should have never made it into production. It could all be copped up to an accident. It will really depend on whether the company wants scapegoats or wants to cop it up to institutional incompetence. Unless they find documentation that says, "TODO: Remove this test code" or "Bwahahaha cheat the emissions test!" how are they going to prove it either way?

        • Re:Nail everyone? (Score:5, Insightful)

          by mbone ( 558574 ) on Thursday September 24, 2015 @09:53AM (#50589383)

          Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

          In other words, the engineers who actually did the code may not have known the real purpose of what they were doing. Somebody knew, of course, but they may be harder to track down.

          • Re:Nail everyone? (Score:5, Interesting)

            by Wrath0fb0b ( 302444 ) on Thursday September 24, 2015 @11:28AM (#50590183)

            Yeah. I would not be too surprised if at some level in the organization this was sold as a debugging or trouble shooting measure, or some other benign reason was given for branching on detection of emissions tests.

            Folks, you have to branch on emissions (and dyno) tests in the ECU solely because otherwise the safety side of things will bring everything to a halt. The most obvious reason (assume FWD) is that if the front wheels are going 65MPH and the rear wheels are going 0MPH, the traction control system is going to have a major freak out and say "HOLY CRAP WE ARE SKIDDING OUT OF CONTROL BRAKE FRONT UNTIL THE DIFFERENCE IS LESSENED". The procedure [hellcat.org] to enter dyno mode [bimmerforums.com] is not itself a secret.

            So it's not a secret and it's not illegal for the ECU to detect and behave differently during the test, and everyone would know about this above-board feature. The secret-and-illegal part is modifying the behavior of things measured by the test while the test is running. That is a lot easier to keep secret and requires a lot less involvement from teams directly outside the module that is responsible for emissions.

            [ Source: Tuner people that dyno fancy cars and have to solve these sort of issues. ]

            • Bump the parent

              Yes - all the ECUs do this - and my informants tell me they all cheat - this is really about selective prosecution because VW is non UAW.

               

        • Re:Nail everyone? (Score:5, Interesting)

          by PaulRivers10 ( 4110595 ) on Thursday September 24, 2015 @10:05AM (#50589499)
          Yeah, I worked for a large bank. I was told my task was to implement tax calculation code in javascript, so it would update on the page immediately. While I balked at the request, it was made pretty clear that either I do it or I would get fired. I was not given a javascript library that knew how to handle financial values. Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors. I happened to know about them but was not given any instructions that they were a problem. I wasn't given any instructions on how to make sure they didn't cause issues. As far as I know no more than basic testing was done on the code. I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that. You get fired now, or you implement something dubious - what do you choose?
          • Re:Nail everyone? (Score:5, Insightful)

            by Archangel Michael ( 180766 ) on Thursday September 24, 2015 @10:31AM (#50589687) Journal

            You get fired now, or you implement something dubious - what do you choose?

            You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

            One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

            Even if you have to write it ... "Per our conversation regarding _______ I am doing _______ at your request. Please let me know if you change your mind". In civil court, all you need is 50+%. It doesn't take much to get to 50+%. Self documentation is perfectly acceptable.

            It has saved my bacon a number of times.

            • Re: (Score:3, Insightful)

              by Anonymous Coward

              Yes and in the mean time you lose your house, car, and your kids go hungry. And the woman that would love you till death do you apart? She meant financial death. She will leave you for putting your family's financial livelihood in jeopardy to do some half-ass ethics play. You may get some money from the settlement, but the whole thing will now be part of public record. Good luck getting hired ANYWHERE else.

              It's easy to be an idealist on paper, especially if you have nothing to lose. Real life is a tad

              • Actually it's quite simple in real life and in most sane countries in the world (the USA excluded by this statement) wrongful dismissal that needs to be settled is either settled very quickly (by tribunal) with financial compensation and re-employment or a buyout of the employee, or at the end of the court case you can retire with your new found wealth.

                That's how it happens in real life in most of the western world where employees are protected by laws not bought by a corporation.

            • You get fired now, or you implement something dubious - what do you choose?

              You get fired. Then you sue for wrongful termination. Then you expose the company in court (public record) about how shitty they are threatening you with termination (you have proof? right?) for doing something dubious.

              Dubious and wrong aren't the same thing and the OP didn't explain further. He's a programmer and his employer asked him to write some tax calculation code. Either he can research the math and write some code or he can't. If he can't or refuses, getting fired isn't "wrongful termination", though it would be a dick move to fire him if he's simply not up to the task w/o affording him a chance to learn. Documenting his concerns is a good CYA idea in case anything goes south. Now, if his employer asks him to w

            • Re:Nail everyone? (Score:5, Interesting)

              by Anonymous Coward on Thursday September 24, 2015 @11:51AM (#50590345)

              IANAL, but this is what I heard from an employment lawyer (paraphrasing): As long as you wrote and sent the e-mail, the onus is on the employer to show that they have replied to it in a reasonable fashion. Courts tend to throw out "I didn't see it" excuses and expect replies. And even if they replied verbally, oh well, too bad, they should've done so in writing.

            • One thing I have learned is you always say "Can I get that in writing?". This alone stops a huge number of stupid decisions, especially when you're protesting.

              That worked on one of my tasks where I was asked to do something counter to standards. After I tried to argue them down for some time, my coworker (I wish I could take credit but I've made use of this since then) simply asked for them to send us an e-mail stating what we were being asked to do. At that point, the other person's manager told him to stop asking us to do it the wrong way.

          • by LWATCDR ( 28044 )

            " I was not given a javascript library that knew how to handle financial values."
            You're a programer so write one.
            "Javascript doesn't support integer-only values, so you're doing financial calculations with floating point roundoffs and errors. "
            You're a programer so write one.
            Really just make a fixed point object or even a BCD object for the project.

          • by sjbe ( 173966 ) on Thursday September 24, 2015 @10:32AM (#50589701)

            I did get an email verifying that I had questioned it, but then I found out that all our emails are automatically deleted after 6 months or something like that.

            Nothing prevents you from printing emails of instructions to implement dubious decisions. I've done this from time to time just to protect myself when I worked at a large company.

            You get fired now, or you implement something dubious - what do you choose?

            If it is clearly illegal or will be very likely to cause major problems then you should seriously consider walking. If it isn't so clear then you get them to document their instructions to you and you keep a copy (print if you have to) for your records to cover your ass should it be a problem down the line. Make sure you document your objections and make it clear that you have taken every reasonable effort to ensure that what you are doing is legal. If the decision is merely dumb but legal, same thing but don't worry so much about ensuring legality.

          • Just drop that round off in to some other account.

    • Hmm...my big question is...how can the avg person easily do this...so as to now be able to mod his car as he wishes and have it pass inspection when it has to.

      :)

      Fortunately I've never lived in a state that requires emissions *sniff* tests, but I understand it is a PITA in some states to put on even a simple aftermarket exhaust system for performance.

  • Aw... (Score:5, Insightful)

    by 93 Escort Wagon ( 326346 ) on Thursday September 24, 2015 @09:30AM (#50589203)

    It's cute how he thinks no one thought about this and sanitized the audit trail. I'm sure he also thinks his 4096-bit disk encryption thwarts even the most determined ne'er-do-wells.

    • Re:Aw... (Score:5, Interesting)

      by Anonymous Coward on Thursday September 24, 2015 @10:05AM (#50589509)

      Why would anyone want to hide evidence? All the people seem to operate under the impression that this is some kind of big revelation, when in fact everyone in the industry knows that this is happening all over the place. This wasn't actually big news that Volkswagen is cheating, the big news is that some authority is finally taking the rules seriously, when for years both the european and US authorities did their best to look the other way when the evidence was shoved in their face. And if no one cares, why would you want to hide it?

      • Re: Aw... (Score:5, Informative)

        by Anonymous Coward on Thursday September 24, 2015 @10:41AM (#50589785)

        Definitely been going on for at least 17 years, and many other companies besides VW are getting caught.

        "On October 22, 1998, the Department of Justice and the Environmental Protection Agency announced an $83.4 million total penalty against diesel manufacturers, the largest civil penalty ever for violation of environmental law...The seven companies sold 1.3 million heavy duty diesel engines containing illegal "defeat devices," which allow an engine to pass the EPA emissions test, but then turn off emission controls during highway driving. As a result, these engines emit up to three times the current level for NOx a harmful air pollutant."

        http://www2.epa.gov/enforcement/detroit-diesel-corporation-diesel-engine-settlement

        You can find many more examples of enforcement at:

        http://www2.epa.gov/enforcement/clean-air-act-vehicle-and-engine-enforcement-case-resolutions

        (AC because I work in the industry.)

    • Re:Aw... (Score:5, Interesting)

      by Tablizer ( 95088 ) on Thursday September 24, 2015 @10:55AM (#50589891) Journal

      Based on my experience with slimebag orgs and managers, it may have gone something like this:

      A manager(s) asks for a software switch to deactivate the "clean" mode and also detect when smog testing is being done "in order to study and track resources devoted to environmental issues and make sure we understand and comply with the smog testing procedures."

      Then a personal visit happens where key manager(s) ask the top-ranking technician to leave the bypass-on-test feature "on" in production. No paper trail. Experienced slimebags don't put such commands into writing.

      When the IT lead later reveals "Executive X told me to in person", there's no written trail. It's one person's word against another's.

      Sure, the IT lead is probably suspicious of the request, but when the big bosses tell you to do something, it's comply or hit the road.

      I was once asked to cheat a client over database scalability. It was shortly after the dot-com crash, and knowing the market was really tight in Calif. and having a young family, it was a really difficult situation to be in. I won't go into the details here, but I was sick to my stomach over it. The experience made me more progressive.

    • by orlanz ( 882574 )

      If there is no audit trail, shouldn't the buck stop at the top? The roles, job titles, audits, change management controls, financial approvals, etc are all business processes designed to reallocate the default responsibility who is either one of the big Cs or the board of directors. Without the approvals, audits, or whatnot the responsibility goes to the person who can't push further down. And it can't be a he said, she said thing. Need documentation.

      I understand this is not how it works in the real wor

  • by rekoil ( 168689 ) on Thursday September 24, 2015 @09:33AM (#50589229)

    I *highly* doubt it was a single line of code. To toggle the car's "EPA Cheat" mode, maybe, but by all accounts, the system used a variety of inputs to detect artificial driving conditions (including, apparently, barometer data), as well as needing code to define what engine parameters to change once the mode was entered.

    • by ZeroPly ( 881915 ) on Thursday September 24, 2015 @09:36AM (#50589251)
      On the other hand, the code could be in Java. Those programmers are so verbose, all you have to do is search for the cheatOnEmissionsWhileRunningEPATest() functions.
    • It's been a while since I watched my car being tested but do they hook up the car to a computer terminal of some sort? Could those be used to trigger test mode?
      • by Anonymous Coward on Thursday September 24, 2015 @09:49AM (#50589359)

        Sometimes but not always. This cheated the test both with and without a computer. It was instead detecting when it was on a rolling road. Emissions tests are always done with the car stationary but the wheels moving, and that'll be what the software was detecting.

        From the sound it of it wasn't actually putting the car into a special 'mode'. It was turning on all the measures to reduce emissions. When the car was on a real road it was turning them off to get better performance and fuel consumption at the expense of emissions. So it sounds like the car does technically meet the regulations, but ignores them when it's on the road. So expect the recall to turn them on at all times, which'll mean you don't need the car/engine replaced, but will mean you pay more at the pump and see your car's less nippy than it was before.

        • by cayenne8 ( 626475 ) on Thursday September 24, 2015 @09:55AM (#50589407) Homepage Journal

          When the car was on a real road it was turning them off to get better performance and fuel consumption at the expense of emissions.

          Not sure why VW stock has tanked then...personally, this sounds like a very GOOD reason to go buy a VW car now...knowing when it comes to it, that it will really perform well!!

          • by OverlordQ ( 264228 ) on Thursday September 24, 2015 @09:59AM (#50589449) Journal

            Or, "If they're cheating on this, what other things did they cheat on?"

          • by N1AK ( 864906 )
            Because VW are going to recall hundreds of thousands of cars, stick them permanently in test mode with emission stuff switched on. There will then be a massive class action as people who bought a VW expecting x horsepower and y MPG demand compensation for getting considerably lower performance. This will cost them billions, and that's before the multi-billion dollar fine piled on top.
      • by sjbe ( 173966 ) on Thursday September 24, 2015 @09:59AM (#50589437)

        It's been a while since I watched my car being tested but do they hook up the car to a computer terminal of some sort? Could those be used to trigger test mode?

        The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.

        Basically they wanted to avoid the cost of installing a urea injection system [wikipedia.org] so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.

        Here's a good article about what is known so far:

        http://www.msn.com/en-us/autos... [msn.com]

        • by Thelasko ( 1196535 ) on Thursday September 24, 2015 @10:29AM (#50589677) Journal

          The test mode was triggered by monitoring which wheels were turning, position of the steering wheel, etc.

          Basically they wanted to avoid the cost of installing a urea injection system [wikipedia.org] so they cheated instead. Honda engineers were reported to be perplexed about how they managed to do this miraculous feat of engineering.

          As a diesel emissions engineer, I was always fascinated by how Volkswagen was able to do what they claimed. I had tried to make their technology work, it's extremely difficult. SCR is much simpler and more economical.

          I had thought for a while that other companies had some secret information that my team was missing to get such good performance. Now I think other companies are just cheating.

    • $ behave -d | grep "scenarios passed" | cut -d, -f4 | sed -e 's/^[[:space:]]*//' | sed 's/untested/scenarios/g'

    • by jandrese ( 485 ) <kensama@vt.edu> on Thursday September 24, 2015 @09:38AM (#50589269) Homepage Journal
      Supposedly the "cheat mode" is an extension of the "testing mode", where the car knows it is running on a Dyno because one set of wheels is turning at a high RPM and the other set are stationary. For a car with traction control this is normally a freakout event so they have to check for it and make sure not to go crazy just because the machine is strapped into a test harness. Once you have the otherwise required detection code in there, adding a single line to fully open the EGR valve when in that mode would be a piece of cake.
      • by Chris Mattern ( 191822 ) on Thursday September 24, 2015 @10:03AM (#50589493)

        Yes, but people run these cars on dynos all the time for performance figures, and they all saw the standard performance figures. They fixed it so that the cheat mode would kick in when EPA testing was done on a dyno, and only when EPA testing was done on a dyno. It's doable, thanks to the EPA's rigid testing protocol, but it wouldn't have been simple.

      • by AmiMoJo ( 196126 ) on Thursday September 24, 2015 @10:14AM (#50589583) Homepage Journal

        According to news reports (BBC etc.) it used a combination of inputs. Steering wheel position, barometric pressure variation over time, rate of acceleration, speed, g sensor stability etc.

        Also, to enable the cheat mode the engine would have had to load a different set of operating parameters. Those parameters must be stored somewhere, and doubtless constitute more than a single line of code.

        Some thought must have gone into this fraud.

        • Also, to enable the cheat mode the engine would have had to load a different set of operating parameters. Those parameters must be stored somewhere, and doubtless constitute more than a single line of code.

          Some thought must have gone into this fraud.

          You don't just punch in a guess at those operating parameters either. Lots of data gathering and evaluation was done to get them to meet requirements. This was a decision at the program (I'm not talking software) level. A budget was approved to design, test, and implement this strategy. There was no lone gunman.

    • There are lots of legitimate reasons to have different 'states' of operation like detecting if the car is running and nobody is sitting in drivers seat.

    • Most rolling roads don't spin the non powered wheels, so if the powered wheels are spinning and the others are stationary for any length of time its a good bet its having some sort of test. Obviously this isn't going to work with 4WD however.

    • He never said how long that one line actually is... you could *technically* write an entire program in one line.
  • by avandesande ( 143899 ) on Thursday September 24, 2015 @09:35AM (#50589239) Journal

    "However, not all companies follow detailed auditing processes. The primary reason, Kaul said, is the speed at which software is being released to the marketplace. It necessitates an "agile approach," resulting in millions of lines of code being worked on and checked into production every minute."

    • Re: (Score:3, Informative)

      by Black Parrot ( 19622 )

      You'd have to have a *lot* of programmers to be checking in millions of lines of code every minute.

      And I'm not sure *any* rate of check-ins would justify not using a version control system.

    • by n1ywb ( 555767 )
      That's true at a lot of software companies but it is almost certainly completely untrue when it comes to an embedded engine control unit at a major automotive manufacturer. That sort of development is typically slow, methodical, and rigorous, with extensive pre-release testing. And the team is probably pretty small, I doubt there are more than a dozen engineers working on that, and probably a handful of key guys write most of the code. And I can guarandamntee that their code churn is nothing like "millions
  • I wonder whether someone actually gave the order to implement a 'test defeat device' or they just started to optimize the engine to comply with regulation and to pass the test and then they went too far.
    • by sjbe ( 173966 ) on Thursday September 24, 2015 @10:09AM (#50589545)

      I wonder whether someone actually gave the order to implement a 'test defeat device' or they just started to optimize the engine to comply with regulation and to pass the test and then they went too far.

      Someone in management made the decision to not install a urea injection system which is necessary to keep emissions to legal levels but costs a lot of money. Reportedly something like $400/vehicle. So it seems pretty clear that their "solution" to the problem was simply to cheat. This wasn't a case of optimization gone awry. They flat out knew what they were doing and went ahead with it anyway. As soon as they made the decision to not install urea injection, they effectively decided to cheat at that time because they were asking for the technologically impossible. There is no way they didn't know that their decision to leave off such a key piece of equipment would not result in unacceptable emissions. The engineers at VW aren't dumb. The decision was made for financial reasons (not surprising) but was aided and abetted by a bunch of engineers that should have known better.

      The only real question seems to be who made the decision and who was responsible for executing it and covering it up.

  • by Ihlosi ( 895663 ) on Thursday September 24, 2015 @09:38AM (#50589275)
    They should have studied the entries to the underhanded C contest ... to make things a little less obvious.

    Then again, something similar might make a nice contest topic.

  • by lesincompetent ( 2836253 ) on Thursday September 24, 2015 @09:40AM (#50589289)
    Someone should have leaked this a looong time ago. Perhaps some dev, why not.
    Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln!
    Did VW really think it could get away with this indefinitely?
    Fucking corporate morons...
    • Furthermore, another question popped up in my mind: how come none of the competitors discovered this in their testing of VW cars?
      Perhaps they're doing this too so they didn't want to blow the lid?
      • by Captain Hook ( 923766 ) on Thursday September 24, 2015 @10:22AM (#50589625)

        The way this was discovered (*1) was by an independent university group taking purchased vehicles and connecting it up with sensors and running it over real roads in real traffic conditions over long periods of time and comparing it to the rolling road test results. It's not an astronomical cost but if you are just looking for basical emission data then there are much simpler methods (namely a rolling road).

        *1 - assuming this wasn't a case of parallel construction and the real road test data is just collecting evidence for what somebody already knew was happening.

    • Did VW really think it could get away with this indefinitely?

      Nope. Just long enough for the authorizing executives to get their bonuses.

    • Hell it would have saved VW a lot of money! Think about recalling 1mln cars instead of 11mln!

      I know if I had one of these cars, I'd certainly NOT be taking it in for the recall work.

      Getting the better mileage and performance out of my car like this sounds like a benefit I'd want to keep!!

  • by judoguy ( 534886 ) on Thursday September 24, 2015 @09:43AM (#50589311) Homepage
    I haven't found out if the normal driving emissions are actually"bad" or just fall foul of U.S. automaker protectionist lobbying.

    No flame here, just wondering. In my travels to Europe I haven't found them to be any worse pollution wise than American cities. Are these cars really that bad physically or are we talking goofy government crap?

    Just asking.

  • Not so outlandish (Score:5, Interesting)

    by vikingpower ( 768921 ) on Thursday September 24, 2015 @09:43AM (#50589319) Homepage Journal
    I have been teaching UML, modeling and systems architecting at several companies that directly supply to the German car industry ( especially to Volkswagen and BMW ). It is the car makers themselves that impose rigid rules and constraints upon software traceability and configuration management. So the idea of

    "software dev/test audit trails are almost certain to pinpoint who embedded the code and who authorized it"

    is not that outlandish, and following such audit trails may well lead to (at least some of) the culprits.
  • May be not (Score:5, Funny)

    by sshir ( 623215 ) on Thursday September 24, 2015 @09:45AM (#50589329)
    If I were doing it I would have placed "// FIXME DEBUG" on that line of code. Like it was an internal testing mode which wasn't switched off, by accident of course.
  • EPA Cheat Code: Up, Up, Down, Down, Left, Right, Left, Right, B, A

  • I would really like to see VW's source code, to see if they took the care of plausible deniability. There was recently a nice contest over here [thedailywtf.com].
  • I envision a scenario where as soon as the shit hit the fan on this, somebody really high up in the company who was in the know on this went into the necessary files and altered the documentation so that they were not incriminated.
  • by 140Mandak262Jamuna ( 970587 ) on Thursday September 24, 2015 @10:03AM (#50589487) Journal
    Most of the actual codes that disabled the emission controls would have legitimate use during development cycle. Typically there will be a switch to put the engine in a baseline test mode, to gather baseline data. The test engineers would turn it all off, get a sanity check baseline runs made. Then they will turn on various switches and turn various knobs to fine tune the best values trading off power and efficiency to reduce emissions. So the actual code turning off emission control is not a big deal.

    What would be a big deal however is the code that detects whether the car is in test bench or on the road. Apparently it uses steering input and other such details. So that code block is the interesting part. Proper audit of the code changes, pull request authorizations would nail the engineer who actually did the dirty deed. But would there be code review meeting/minutes, comments fingering higher management?

    This scandal will have some salutary effect in engineers who manage code, they would refuse to merge or pull such cheating code changes because it would leave their fingerprints indelibly for ever. They might even add comments in code covering their tails fingering the actual perp in the higher management.

  • by funwithBSD ( 245349 ) on Thursday September 24, 2015 @12:04PM (#50590443)

    I worked for a Small software house that made SAP type ERP software before SAP ate the majority of the market. This was 1998 or so...

    We had a customer come to us and ask for certain modifications. Then a few more. Then a few more.

    Not unusual, we made a lot of money from change orders. So the first few were done. All were acceptable in the Generally Accepted Accounting Practices guidelines.

    Somewhere along the line the GAAP accountant realized that this last modification set would, taken in combination with all the other mods, make a check disappear from the system and become untraceable.
    We refused to do it, and the customer dropped the product, saying we were too hard to deal with. A million+ of revenue were lost, no small amount for the company.

    That customer? MCI Worldcom.

    They clearly had picked apart the source code and found the edge case that triggered the behavior. I had left the company before MCI blew up, but my understanding is that they were called to give testimony/evidence in the trial.

    This could be the same thing, a series of unrelated changes that trigger a diagnostic mode when hooked up to the test equipment.
    If so, it would be very hard to trace who made the ultimate decision to do this, as it might be spread across many teams working independently.

You know you've landed gear-up when it takes full power to taxi.

Working...