Overeager Compilers Can Open Security Holes In Your Code 199
jfruh writes: "Creators of compilers are in an arms race to improve performance. But according to a presentation at this week's annual USENIX conference, those performance boosts can undermine your code's security. For instance, a compiler might find a subroutine that checks a huge bound of memory beyond what's allocated to the program, decide it's an error, and eliminate it from the compiled machine code — even though it's a necessary defense against buffer overflow attacks."
old news from decades ago (Score:5, Insightful)
well known for decades that optimizing compilers can produce bugs, security holes, code that doesn't work at all, etc.
Re:old news from decades ago (Score:5, Insightful)
Re: (Score:2)
ah, disgruntled and curmudgeonly old coot hand writing assembly from pseudo-code outline. Yeah, done that
Re: (Score:2)
Re: (Score:2)
*cough* Microsoft .net
Re:old news from decades ago (Score:5, Insightful)
Or rather, that optimizing compilers can expose bugs in buggy code that weren't revealed by naive translation.
Re: (Score:3)
Re: (Score:3, Interesting)
Re: (Score:3)
Re: (Score:2)
I've always preferred inline assembly or linked asm routines for tricky bits, but the problem then is it's not portable.
Re: (Score:2)
I don't think any language is portable at that level, so you may as well use asm (my preference is linked instead of inline to ensure a clean and simple abstraction.) Every processor has different math status registers and different math instruction capabilities. I'm not sure how C can express these things.
Re: (Score:3)
Indeed; the compiler's even allowed to assume signed integer overflow doesn't happen, which is where you get into trouble. Yet we have this perfectly good mechanism for detecting integer overflow (condition codes) and no way to reach them from high level languages (C isn't unique in this respect)
Re: (Score:2)
That's an OLD problem. When I was using FORTRAN 66, I had to put a sentinel card at the end of the data I was reading. (Actually, I usually used a nonstandard extension.) The OS knew when my data was all read, but there was no standard way to let FORTRAN know.
Re: (Score:2)
Which is where a compiler should actually issue a warning. Then the programmer, if deciding the code really is necessary that way, adds some pragmas, attributes, or typecasts, and then double checks the compiler output afterwards. I agree though that such cases are sometimes very useful, however a compiler must never optimize this code away silently.
Re: (Score:2)
Arguably, it's a bug in the standard. It defies the principle of least astonishment for a procedural language.
Re: (Score:2)
If you don't understand that it is the standard itself where the astonishment occurs, you are over your head in this discussion.
Re: (Score:2)
The big problem is the extent of undefined behavior. Undefined signed integer overflow made a great deal of sense when C was being designed: some computers would fault on arithmetic overflow, while some would wrap around, and specifying one behavior would have made C run badly on a computer doing the other. (In retrospect, calling it "implementation-defined" would have worked better.) Probably the best use for "unsigned" in C and C++ is to avoid arithmetic overflow, since that's completely defined by t
Re:old news from decades ago (Score:5, Insightful)
Re:old news from decades ago (Score:4, Insightful)
Because it worked in debug mode (which generally has optimizations off)?
Because it was tested on a compiler without this bug? The people writing the memory library is usually not the people writing the app that uses it.
Similarly, it was tested on the same compiler, but with different compiler flags?
Because that optimization didn't exist in the version of the compiler it was tested on?
Because the test app had some code that made the compiler decide not to apply the optimzation?
Life is messy. Testing doesn't catch everything.
Re: (Score:3)
If you're testing only in debug mode, you're doing it wrong. Do your customers run debug? No? Then all your tests must run retail or you fail.
Is this a compiler bug? Doubtful. Chances are it's code that isn't standard that was getting away with its non-standard behavior until the compiler started enforcing this bit.
Compiler flags? Again, test the binaries your users will run.
Testing can catch a lot, if not half-assed.
Re: (Score:2)
And if you're writing a library, you don't know what compiler much less what flags the user will use. Are you willing to pay several thousand dollars for seats for obscure compilers?
Yeah, you're an idiot.
Re: (Score:2)
Re: (Score:2)
If you're writing a library that messes around in the dark corners of C, you should know exactly what you're doing WRT the standard. And what oddball compilers are still in use in an environment where people are consuming open source libraries? Obviously, you want to cover GCC, clang/llvm, VS, and maybe Intel yourself. For the rest, as the sibling post says, libraries should come with unit tests in the modern world,
Re: (Score:2)
Most of the embedded world? The embedded world heavily uses compilers like RVDS. And nobody is going to bother running the unit tests. In addition, its damn hard to write unit tests that test something that should fail in a stack dump.
Re: (Score:3)
That would be, you know, sane? And competent? Cannot have that in somebody that does software...
There are also nice compiler directives that are used to switch off optimization for some functions.
Re: (Score:2)
Because of TINSTAAFL.
Just because you want extra safety checks around _your_ code does not imply that I can afford the performance _penalty_.
C is a mid level language because it tries to have zero over-head. If you want extra checking done YOU tell the compiler you explicitly so that it doesn't effect everyone's run time performance implicitly.
While I agree it would be extremely useful to be ability to have "safe arrays", and even tag data with "volatile" to tell the compiler "Hey this is important due to
Re:old news from decades ago (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
This is assuming you can write 100% bug-free code. You can't, for any realistically sized project.
Re: (Score:2)
You can write a safe library by adding provably correct constructs to a zero-overhead one. You can write bug-free code for small pieces of code, and you can combine these pieces to get code that almost certainly has bugs but is known not to have certain specific bugs.
Re: (Score:2)
TINSTAAFL? Mixing idioms much?
The famous phrase (quite deliberately using a non-mainstream dialect) is "there ain't no such thing as a free lunch": TANSTAAFL,
If you wish to recast it in a standard idiom (for reasons I cannot divine) then you should go with "there isn't any such thing as a free lucnh": TIASTAAFL. But this is pure pedantry.
Re: (Score:2)
ain't is not proper English
How about focusing on the actual issue instead pointless pedantic non-issues please.
Re: (Score:2)
Though technically the compiler would be incorrect by removing "unnecessary" code when it actually is necessary. The thing is, if the compiler is good enough to detect that the code is checking out of bounds when it doesn't need to, then the compiler was able to logically infer that it was impossible to write code out of bounds, which means it should be good enough to print out a warning when the code does have possibility of buffer overlows.
The examples given in the articles listed are things that I would
Re: (Score:2)
Your statement is perfectly true.
Unfortunately, faulty compilers are more than just hypothetical. But besides that...
Even with a perfect compiler, the knowledge it takes to avoid these 'equal semantics' traps is beyond most experienced programmers (no matter how much our egos say otherwise.) But otherwise, I totally agree that most 'optimizer bugs' I've seen are just lack of understanding by the programmer. The compiler rightfully saw its solution as equivalent to what the source code stated.
I think ther
Unsable Code, again (Score:5, Informative)
Re: (Score:2)
I blame both. When a compiler discovers code with undefined behavior in the language, then the compiler should issue a warning rather than taking it as an opportunity to silently perform some other optimizations. In other words, the compiler KNOWS the code is faulty, and it can't perform the extra optimizations without first knowing this. Thus I absolutely blame the compiler writers here.
Ie, if the compiler sees "y = 0; z = x / y;" then it should warn about division by zero. Instead if the compiler thi
Re: (Score:2)
How about when the computer sees "z = x / y;"? Should the compiler warn of possible undefined behavior if it can't prove that y can't be zero?
Re: (Score:3)
Actually, that makes sense.
Suppose we want to check for buffer overflows in C++, so we allocate a stretch of memory beyond the buffer, zero it out, and check it to see if it's zeros later. Depending on how the buffer is used, it may be that a buffer overflow would require undefined behavior, such as accessing memory beyond the limits of a data structure. An optimizing compiler might figure that out. In event of undefined behavior, anything the compiler does is conforming, and in event of no undefined
Re: (Score:2)
stretch of memory beyond the buffer, zero it out, and check it to see if it's zeros later.
I don't remember the correct term for such regions, but nulls are a very bad sequence to put there, since... they can't be distinguished from actual, purposeful zeros.
Better to initialize the area with something like 0xDEADBEEF.
Re: (Score:3)
Compilers remove vastly more code than you seem to think, when optimizing. They always have - this isn't new. You wouldn't want all that spam as warnings, for sure (if you build with warnings as error, as one ought to do, you'd never build anything).
UNISEX conference (Score:5, Funny)
But according to a presentation at this week's annual UNISEX conference
Re: (Score:2)
I was wondering why my compiler was generating warnings about traps. I thought it meant something about catching buffer overflow conditions.
Re: (Score:2)
Re: (Score:2)
Hang on, this compilation is going to be great! I'm so excited! Let me just start parsing those header files, and... Oh, we're done. Apparently I've optimized away almost all your code. Thank you and good night.
Re: (Score:2)
Complete nonsense.... (Score:3, Insightful)
Any code removal by the compiler can be prevented by correctly
coding the code with volatile (in C) or its equivalent.
Re: (Score:2, Informative)
Except not, so now we have explicit_bzero()
Re: (Score:2, Insightful)
Any code removal by the compiler can be prevented by correctly coding the code with volatile (in C) or its equivalent.
Knowing that the code will be removed by the compiler is paramount to using the volatile keyword.
That requires knowing a lot more about what the compiler is going to do then one should presume. The developer should not have to have foreknowledge of what compiler optimizations someone might enable in the future, especially as those optimizations might not even be known in the present.
The norm
Re: (Score:2)
---
I can't offhand think of many situations where I could say with any degree of certainty that if something read or wrote to memory externally that it wouldn't matter, and it would rarely be the best use of my time to try an establish it... so really... mark everything volatile all the time.
Clearly THAT isn't right.
---
Yeah, that would be a poor design. You use volatile when you need to. That's the rule. You just need to work out when you need to.
> If you do not expect the memory to be written to or rea
Re: (Score:2)
I'm not sure you typed that in right, or maybe you don't understand what volatile means or when to use it.
No I typed it in right. That's the point. To guard against security flaws, you need to specify volatile in situations where you explicitly expect it not to be volatile; so that any sanity checks you put in place don't get optimized out as redundant/dead code.
For security you effectively have to assume all memory is volatile, and that it might be changed when it shouldn't be.
For a contrived example,
stati
Re: (Score:3)
Except it's just as vulnerable either way. Your sanity check never executes even if x and y are volatile, since the buffer overrun in somestuff() already transferred control to pwned() when somestuff() tried to return. All your sanity check does is give you a false sense of security.
Re: (Score:2)
since the buffer overrun in somestuff() already transferred control to pwned() when somestuff() tried to return
That is only -one- failure mode, not all buffer overruns will let you overwrite to the instruction pointer to transfer control to your code.
Security is layered.
Yes, and that problem is not solvable in C, no matter how many sanity checks you litter your code with. As soon as you execute any code from an untrusted source, you don't have any security. Any error puts the system into an undefined state,
Re: (Score:2)
Except that "volatile" means that the memory might be accessed through methods other than the program, which is the exact sort of thing we want to test for. In C++, "volatile" means that all memory accesses must be performed in the given order, and in proper order with other volatile memory accesses and calls to I/O routines. This removes a lot of optimization possibilities, which is why we'd generally rather not call variables "volatile". For a buffer overflow test, we know that the buffer can't be cha
Re: (Score:2)
Except that "volatile" means that the memory might be accessed through methods other than the program,
Exactly right.
The trouble is in the case of a bug or vulnerability, even non-volatile memory might be accessed/updated when its not supposed to be. And any sanity checks or range changes or bounds checks you might write to try and close the potential holes require the non-volatile memory be marked volatile to prevent the compiler optimizing out the checks.
See my other reply for a contrived example of what I
Re: (Score:2)
Yup. I was thinking of sections of memory allocated strictly to stay zeroed out and then scanned, presumably part of a larger allocation and put there by placement new. Range and bounds checks will not be optimized out. If you have code that says "make sure i is a valid subscript, and if it is return the array value, otherwise do something else", that's completely conforming and will be compiled normally. (Code that says "Return the array value, oh, and by the way, check i" is subject to having the che
Re: (Score:2)
In other words, these vulnerabilities don't rise from compiler optimizations but from programmer errors. They only relate to compiler optimizations insofar that those optimizations interfere with ad-hock attempts to get managed language behaviour in an unmanaged language. Which is
Re: (Score:2)
either write code that can be checked to be secure by static analysis tools
Which just catches a limited subset of vulnerabilities.
or switch to a real managed language.
Because you can't write exploitable code in a managed language?
Re: (Score:2)
Knowing that the code is being removed by the compiler also means that you know that your compiler is broken. The compiler should have issued a warning pointing out that the code is using undefined behavior, not take that undefined behavior as a lame excuse to try some more optimizations. Of course the C/C++ standards do not dictate that warnings must be issued in these cases.
The article itself from what I saw, said nothing about security related vulnerabilities, and nothing about buffer overflow checking
Re: (Score:2)
The compiler should have issued a warning pointing out that the code is using undefined behavior
Not all code the compiler removes is using undefined behaviour. I am not talking about undefined behaviors being 'optimized' away.
The article itself
Yeah, we've sort of gone off track vis a vis the original article. The scope of issues the original article talks about is smaller, and I agree with you fully that the compilers should be issuing warnings for the particular scenarios they are talking about.
Re: (Score:2)
The way I learned about volatile is that it is to be used for memory that is modified out of the scope in which it is used.
What about memory that isn't supposed to be modified out of the scope in which it is used? That's the point. bugs/vulnerabilities that lead to memory that is NOT SUPPOSED to be volatile.
But as soon as you open the can of worms and say, "well... if there's a bug, then maybe it will be modified out of scope somehow... " and then ALL memory is considered volatile.
If you have that understan
Re: (Score:2)
Complete nonsense.... (Score:2)
"...decide it's an error.."
No, it is an "optimizing" compiler not a "correcting" compiler. The optimizer can detect that no language defined semantic will be changed by removing the code, so it does. As others have noted, "volatile" is the fix for this particular coding / compiler blunder. However ill-defined, it is *not an error*.
As for the folks commenting that only C can run in small embedded processors that's hogwash. Huge mainframes of the early ages had smaller memory sizes and ran FORTRAN (now Fortra
Re: (Score:2)
Most made entire classes of C blunders impossible
Don't worry about that! They had their own classes of blunders instead. (Every programming language has a characteristic set of problems that come up, and a set of recommended programming practices that avoid those blunders.)
Re: (Score:2)
Except that the compiler was buggy in the first place by removing such code. Thus it is "over eager". The article is not really about logically correct dead code removal causes security bugs, but about compilers that incorrectly remove code rather than issuing warning or errors in the case of undefined behavior.
Bad summary is bad (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
Actually it's about non-standard-conforming "security" hacks causing unexpected results. If the result of an operation is undefined, the compiler can insert code to summon Cthulhu if it wants to.
If your compiler is doing that, you should choose a different compiler. Summoning elder gods just because signed arithmetic might wrap around is not a good cost/benefit tradeoff!
Re: (Score:2)
....Summoning elder gods just because signed arithmetic might wrap around is not a good cost/benefit tradeoff!
Make that Elder Gods. Respect is essential - we do not wish to arouse their wrath. Nyarlathotep be praised!
Re: (Score:2)
But it calls into questions those compilers. Why did the compilers decide to exploit the unstable code to do more optimization rather than pointing out the unstable code as a warning or error? After all the only way that they could do this extra optimization is if they knew that there were some undefined operations that allowed it to make some further analysis, and such undefined operations might be better treated as defective code.
Although to be fair the examples I saw may have just been oversimplified c
Old news (Score:4, Informative)
I know that at least GCC will get rid of overflow checks if they rely on checking the value after overflow (without any warning), because C defines that overflow on signed integers is undefined. This is even documented. If anything is declared by the language specification as being undefined, expect trouble.
Re: (Score:2)
So why doesn't GCC issue a warning about undefined operations in this case? It certainly issues plenty of warnings in cases that are well defined and not violating any rules.
What "undefined" means here for most compilers is that it will make the best attempt it can under the C rules but the results may vary on different machines. Ie, it will use the underlying machine code for adding two registers, which may wrap around or possibly saturate instead, and the machine may not even be using tw's complement.
Re: (Score:2)
What "undefined" means here for most compilers is that it will make the best attempt it can under the C rules but the results may vary on different machines. Ie, it will use the underlying machine code for adding two registers, which may wrap around or possibly saturate instead, and the machine may not even be using tw's complement.
No, that would be implementation defined behavior.
Misleading summary, slashdot (Score:2, Insightful)
The kinds of checks that compilers eliminate are ones which are incorrectly implemented (depend on undefined behavior) or happen too late (after the undefined behavior already was triggered). The actual article is reasonable— it's about a tool to help detect errors in programs that suffer here. The compilers are not problematic.
What's the would-overflow operator? (Score:2)
Re: (Score:2)
Until you actually need to exploit the undefined behavior. It's somewhat common to have a range of numbers that wraps around rather than distinctly having a smallest and largest value. Ie, indices into a circular buffer for example. You may want a 16-bit value that acts as a sequence number on unacknowledged packets. Eventually it wraps around so that sequence number 1 is actually larger than sequence number 65534. Then you can do operations to see if X is greater than Y in such a system. But most lan
Re: (Score:2)
It's turtles all the way down...
resort to the long long long type?
Re: (Score:2)
Why aren't the compilers problematic? From what I have read the compilers seem to be exploiting the undefined behavior as an opportunity to perform additional optimization, whereas the compiler seems like it should instead warn the user about the undefined behavior.
Maybe it's that the examples that I saw, referenced from the article, are all ones where most basic static analysis tools will flag such code as defective. I haven't seen an example yet where the code is reasonable and does not deserve any comp
Floating point algorithms too (Score:2, Informative)
Compilers can also "optimize" away Kahan summation algorithm. See page 6 of How Futile are Mindless Assessments of Roundoff in Floating-Point Computation [berkeley.edu]
Bad advice (Score:2)
Short of bugs in the compiler's optimizer — and we all know there have been many — the idea that "if the entire code absolutely must stay fully intact, it shouldn't be optimized" is already dangerous.
A compiler conforming to its documentation or standard isn't going to change semantics that have been guaranteed by that document. Those guarantees though are all you have: even without explicit optimization options, a compiler has a lot of freedom in how it implements those semantics. Relying on
Functionally correct, but insecure (Score:5, Insightful)
The classic example of a compiler interfering with intention, opening security holes, is failure to wipe memory.
On a typical embedded system - if there is such a thing (no virtual memory, no paging, no L3 cache, no "secure memory" or vault or whatnot) - you might declare some local (stack-based) storage for plaintext, keys, etc. Then you do your business in the routine, and you return.
The problem is that even though the stack frame has been "destroyed" upon return, the contents of the stack frame are still in memory, they're just not easily accessible. But any college freshman studying computer architecture knows how to get to this memory.
So the routine is modified to wipe the local variables (e.g. array of uint8_t holding a key or whatever...) The problem is that the compiler is smart, and sees that no one reads back from the array after the wiping, so it decides that the observable behavior won't be affected if the wiping operation is elided.
My making these local variables volatile, the compiler will not optimize away the wiping operations.
The point is simply that there are plenty of ways code can be completely "correct" from a functional perspective, but nonetheless terribly insecure. And often the same source code, compiled with different optimization options, has different vulnerabilities.
Re: (Score:2)
If there is external evidence that the compiler needs in order to compile/optimize correctly then it needs to be given to the compiler. Thus the addition of "volatile". Other compilers have a way to add options on the command line, such as indicating that the machine uses strict alignment.
The problem here is however not that there is some strange stuff happening with the program, and that code is being optimized away because the compiler can prove that it serves no purpose given the lack of additional inf
Re: (Score:2)
Unless all the code running on the machine is absolutely type-safe and only allows "safe" reflection then trying to hide sensitive data from other bits of code in your address space is a lost cause. Code modification, emulation, tracing, breakpoint instructions, hardware debugger support, etc. are all viable ways for untrusted code with access to your address space to steal your data.
Wiping memory is only effective for avoiding hot or cold boot attacks against RAM, despite its frequent use for hacking terr
Re: (Score:2)
Your complier constantly removes tons of unneeded code under optimization. It can't possibly be a warning to do this, or you'd drown in them.
Re: (Score:2)
Flatly false.
Good code has plenty of null-checking, for example, most of which a clever compiler can prove is unreachable. It's still completely good code.
Compilers remove code that is provably unreachable on this build. Good defensive coding includes protection against future code changes.
Re: (Score:2)
My approach to the subject (Score:2)
I always insist on a clean compile with the warning level turned up as high as it will go. If the compiler is cool with my code, I have a better chance it will do the right thing with it.
Once I have an application that works I see if it meets performance goals (if any). If it does, I'm done. If it doesn't, profile, find the hot spots, optimize as needed. Compiling an entire application with -O3 is idiotic, and misses the point.
...laura
Re: My approach to the subject (Score:2)
Re: (Score:2)
Suppose your program accesses millions of array elements in performance sensitive areas. Bounds checking would slow down your code by a factor of 2 or more and therefore should be optional (but not non-existent, like in C).
Re: (Score:2)
Sure. Use C++, use std::string and std::vector instead of C-type strings or arrays, and wherever you have brackets you substitute ".at()". Bounds checking guaranteed, and you can globally search for "[" to see if anybody's violating the rules.
Re: (Score:3)
More than that, string and vector are just as fast as the vulnerable C alternatives as long as you pre-allocate the buffers to the expected size. It's not rocket science, but the number of times I've heard "but C++ is too slow, we have to use C" makes me cry. With bounds checking in both (hand-written in C, of course), the speed is the same. Without buffer checking in either, the speed is the same.
Re: (Score:3, Insightful)
C became popular because it was vastly more portable and performant than its predecessors. It still is today. None of those "better" languages that came before it or after it can beat that. And yes, extreme portability does matter when you have 100s of millions if not billions of devices that can't run anything but assembly or C. It's why the people saying that OpenSSL should be written in Java or C# are morons. Care to tell me how that's going to run on a, for example, Linksys WRT54G with only 8 or 16 MB o
Re: (Score:2)
C became popular because it was vastly more portable and performant than its predecessors.
With the exception of Forth. :-)
And yes, extreme portability does matter when you have 100s of millions if not billions of devices that can't run anything but assembly or C.
Or Oberon. Oops, there. I said it.
Care to tell me how that's going to run on a, for example, Linksys WRT54G with only 8 or 16 MB of RAM, 2 to 4 MB of Flash storage and a 125 to 240 mhz MIPS CPU? Yeah, it's not.
Correct me if I'm wrong, but aren't Tektronix oscilloscopes still running embedded Smalltalk these days?
Re: (Score:2)
Re: (Score:3, Insightful)
Well I'd be pretty pissed as well if my pet language was relegated to the graveyard of obscurity by a language that was usable for real work. Dennis Ritchie was a pragmatist who got shit done not some guy wanking over the greatness and purity of the language he created. People to this day are still jealous of that.
Re: (Score:2)
The problem is that most programmers have never had to get their hands dirty doing embedded work. They live in a bubble that ignores all the memory/storage/processing-power constrained devices all around them. OpenSSL, for example, as used in something like DD-WRT would be unusable if it was written in anything but C or possibly C++.
Re: (Score:2)
But can you write for that device in a language that has proper bounds checking built in from the get go?
Re: (Score:2)
Re: (Score:2)
The more things change.... (Score:2)
vaxocentrism [catb.org].
Re: Simple. (Score:2)
Indeed. But that's a different class of problem. Or are compilers optimising constant time comparison routines to not run in constant time these days?