"Side By Side Assemblies" Bring DLL Hell 2.0 433
neutrino38 writes "This is an alert for all developers using Microsoft Visual Studio 2005. At the beginning of January, Microsoft issued a security fix for Visual Studio 2005 forcing the use of new dynamic libraries (DLLs) by all applications compiled with this IDE. Basically, applications compiled with Visual Studio 2005 will not work anymore on an ordinary (non-dev) PC unless the newer DLLs are installed. And we found out that this is true on fully updated PCs. I just posted some details and some suggested fixes." Read below for some more background on Microsoft's so-called "side by side assemblies."
For those unfamiliar with the Microsoft world, native microsoft applications written in C++ rely on dynamic libraries. Two of them are infamous: MSVCRT.DLL and MFCxx.dll. Because of software evolution and security fixes, multiple versions of these DLLs were often present in the system, causing application instability. Where Linux implemented a simple suffix notation on the dynamic libraries, Microsoft created a new beast in 2001: the Side By Side assembly. These are basically DLLs with a companion XML file that identifies them. The XML file contains a digital signature and when the system binds these DLLs dynamically to an application, it checks that the signature of the DLL matches the DLL itself. When everythings runs well, this is pretty transparent. But when issues arise, it becomes excruciatingly difficult to troubleshoot and fix. DLL hell is not over.
Also... (Score:5, Informative)
What might be interesting to note here is that the summary isn't everything there is to side-by-side (SxS) assemblies.
Suppose you're building an application using two DLLs, let's call them A and B. Both depend on a third DLL named C. Now, suppose A uses C version X, and B uses C version Y. You're screwed, right? Not with SxS, since that allows multiple versions of C to be loaded. That's the real added value of SxS.
All this is in theory of course, which as we all know is in theory equal to practice, but in practice is not...
Re:Speaking as a user (Score:4, Informative)
Re:Why don't apps just use their own copy of the . (Score:5, Informative)
That defeats the whole purpose of a DLL anyway. The thought was that you wouldn't have to "reinvent the wheel" and you could reuse code. However, Microsoft's failure to document their operating system's API thoroughly in a public manner led to developers relying on undocumented features that were later changed. Then, those applications needed older versions of those libraries and would install them over the newer versions. This, of course, crashed the newer applications. Ugh.
Re:Speaking as a user (Score:5, Informative)
Re:Also... (Score:5, Informative)
It might surprise you, but Microsoft isn't actually to blame here. Rather, the legions of incompetent programmers that wrote DLLs such as C are to blame. We'd call them idiots, but Microsoft calls them paying customers. Thus prompting them to design SxS and incorporate it in WinXP.
Also, SxS is what made the restyling of the UI (through common controls version 6) technically possible.
Microsoft takes backwards compatibility very seriously.
Re:Speaking as a user (Score:3, Informative)
But the other side is that the OS is massive.
c:\windows alone is 11gb, with \winsxs being around 6gb.
Googling shows that \winsxs is where all these dll versions are being stored.
I haven't got close to 11gb of applications installed (and I've got VS, SQL server + office on here).
Re:Non-issue for actual msdn coders like myself (Score:3, Informative)
YHBT... anyhow, on OS X you can have multiple versions within a framework, and frameworks private to an application. So "it just works."
Packages work beautifully. Anything I need for my app can be bundled right along with it, or in any of the standard directories.
Re:Speaking as a user (Score:2, Informative)
Totally agreed - and additionally, the rigid static linking doesn't work in a plugin environment where each plugin (which is in itself a shared library/dll) brings in its own (shared) dependencies based on the users requirements. Apologies to the OP, but users really shouldn't make demands at this level - there are many reasons why developers and packagers prefer dynamic loading, and likewise it's *their* responsibility to get it right such that users aren't even aware of this level of detail.
Re:Speaking as a user (Score:3, Informative)
Also keep in mind you're looking at a huge RAM footprint as well, since multiple copies of the same library will be loaded per-application, whereas sometimes you can make optimizations for libraries shared across applications with paging magic.
That said, I thought that OS X apps were statically linked (except with OS libs), and thus tend to be large, but reduce this issue since so much functionality is based on system libs. I could be wrong; I don't really ever work with non-linux systems.
Re:Non-issue for actual msdn coders like myself (Score:3, Informative)
Multiple versions per framework is essentially how sxs works.. it's the same idea.
Private frameworks to an application isn't new - you can put DLLs local to a windows app too if you want.. OSX has a global frameworks directory that is similar to the sxs directory, and has the same problems eg. if you're linking to the itunes lib you have to check the one you're writing with is on the target system in the package installer (or install it, but for something like itunes that's probably a bad idea).
Re:Also... (Score:3, Informative)
The rest of your post makes sense (though Unix-style versioning can do the same thing), but this point is wrong. Don't fool yourself or anyone else into thinking you can't modify any application regardless of this stuff. Of course you can: all the SxS stuff might do is make it a touch more difficult.
Re:Speaking as a user (Score:2, Informative)
Additional reading (Score:5, Informative)
Everyone (even Windows programmers) should read Ulrich Drepper's piece on how to write shared libraries [slashdot.org]. (Warning: PDF.) (Likewise, even free software developers should read Raymond Chen's blog [msdn.com].)
Re:Speaking as a user (Score:3, Informative)
Just like in the physical world, bad packaging can fuck up an otherwise good product.
Incidentally, on Fedora, the scenario you describe doesn't occur.
Re:DLL hell never left (Score:3, Informative)
I upgraded my Fallout 3 installation yesterday. After patching, the game wouldn't run, returning some fairly obtuse message about import ordinals [google.com]. So I googled the message, and found out it's because the game now links against a newer version of "Microsoft(R) Games for Windows(TM) Live(TM)" whatever. Note that this wasn't some new patch, it's months old and yet this problem, which must realistically be hitting quite a few users, persists. This isn't something you get via Windows Update either, this is just some obscure 'distributable runtime' crap you should know you need?
So let me repeat that: Super mainstream game on a super mainstream platform (Vista x64), no add-ons, I patch to the latest version and it won't start, nothing is mentioned at the developer's site.
Now I recognize good old Bethesda again. Here's how they'd be able to repro: Fully updated Vista machine, install game from DVD, apply patch, notice it won't fucking run.
I don't normally give much for the 'PC-gaming sucks' choir, but c'mon..
I had the same problem. Only, I run Fallout3 in Linux via WINE and there is apparently no way whatsoever to get xlive.dll to work in WINE. In addition, you do need the latest Fallout3 patches in order to install the expansions. Personally, I found it unacceptable that I would not be able to use any of the expansions merely because someone decided to add functionality that I never asked for, do not need, and will never use.
I found a solution. There is a patch for Fallout3 that removes all Live functionality, and was sufficient to get the game working for me in WINE because it removes the dependency on xlive.dll. I can now use the game with all five expansions (Mothership Zeta was particularly fun). A page describing it can be found here [loyalkng.com] while a direct link to the download is here [mediafire.com].
Re:Non-issue for actual msdn coders like myself (Score:5, Informative)
Oh jesus christ I can't stand how fucking idiotic everything in this discussion is.
What happened: In August, they released a patch to the IDE that required a new version of the shared libraries. Binaries built with this patched version of the IDE/compiler/toolchain will by default require a version of DLLs that aren't installed by default on many systems.
What would have prevented this asshat's blog post: Write a fucking installer that includes the DLLs you use. Thus, when your compiler changes, and your new packages rely on some stuff, magically that 'some stuff' is bundled along with your binary, and everything _still works_. Relying on OTHER packages to have good installers that properly put stuff in to SxS is just idiotic. "Wah, I rely on stuff that I'm specifically told I can't rely on, Microsoft sucks, and SxS is evil!"
No. SxS has been SAVING your ass from updates like this since Windows XP. Surprisingly, you can now NOT recompile your project, and have it still work, even though there's a newer version of DLLs you depend on, because the backwards compatibility problem of these DLLs is just gone. The version YOU want and need and depend on is still there, still usable. Something else on the system that wants the newer version gets it, and all its patched glory. This is actually a security hole, and in THAT case, the DLL you depend on WILL change to fix the security problem. Nothing in the August 2009 update to VS 2005 seems to indicate that this happened, it's only for newly compiled binaries.
So, in summary: The blog poster is an idiot, he hates his user, he's too lazy to write a real installer, he has incompetent developers that don't care about the security of their product, and he asked for help and got 4 completely idiotic responses. Now everyone on slashdot who knows nothing about SxS and the actual problems it causes (and yes there are some) think that it's the same problem we've always had, but worse.
Note: I'm not a Windows developer (Linux developer), and I've not used Windows in well over a year (Mac user). But I've been bitten by SxS in my last job and dealing with VB.Net, back when I didn't know I had to make a real installer for my stuff. I somehow managed to learn to not be an idiot, and the blog poster should too.
Re:Time to question if DLLs are still needed (Score:5, Informative)
Additionally, DLLs can enhance security. If there's a bug in MSVCRT.DLL, Microsoft can patch it and release it as an update. In a static world, every application that had that buggy code compiled into it would have to be rebuilt and redistributed.
In a statically-linked world, every application would need to be rebuilt (on Windows, redistributed) every time an API / ABI change was released, as well as every time a bug was fixed. Furthermore, download sites would either have to release binaries for every API / ABI combination (that's a ton of releases per version) and deal with users downloading the wrong one, or do the open-source model and release the source, forcing the users to build the application on their system and rebuild every API / ABI update. And somehow I don't think the latter solution would fly with the Windows community.
Like other posters have said, Microsoft's solution is actually not a bad one. Allowing multiple DLLs to be loaded simultaneously is not a pretty solution, but it's not a pretty problem that they have to solve, either. Advance with backwards-compatibility in mind as much as it makes sense to, and use SxS DLLs when it doesn't.
Re:The solution is easy (according to TFS) (Score:3, Informative)
Re:dead link (Score:5, Informative)
Oops. Correct link [redhat.com]. (I wish Slashdot would warn about obviously incorrect links.)
Re:Speaking as a user (Score:1, Informative)
and we see the static linked library virus hell, and how microsoft had to write whole-system-wide-scanners to find out if some app had statically linked some jpeg encoding lib that had a flaw in that could be used to attack a system.
statically linked libraries are hell to patch.
Re:Why don't apps just use their own copy of the . (Score:2, Informative)
Re:And by all developers you mean (Score:3, Informative)
This particular issue is C++ only, but that's simply because MS has screwed the way things used to work for unmanaged binaries.
You may still get this issue next time MS issues a patched version of any of the .NET libraries, imagine that your code will only run on .NET 3.5 SP1 patch 1234, then you'll see the exact same problem. I guess we're all used to installing all manner of .NET framework libs through WU so no-one sees this issue so far, but its only a matter of time.
Of course, if binaries used the previous version if the later one was not installed, things would be ok, but that's not the way they're doing things now.
Re:Speaking as a user (Score:5, Informative)
That said, I thought that OS X apps were statically linked (except with OS libs), and thus tend to be large, but reduce this issue since so much functionality is based on system libs. I could be wrong.
Yeah, you are wrong. Mac OS X apps are definitely dynamically linked (the equivalent of so on Mac OS X is dylib). The reason Mac OS X apps can be larger than executables on other platforms is because they often contain machine code for multiple platforms (ppc, x86, x86_64). That only translates to a large footprint in storage.
Now you know.
Re:And by all developers you mean (Score:5, Informative)
But when issues arise, it becomes excruciatingly difficult to troubleshoot and fix.
The version information is written in plain text in the manifest. The files have names based on the version information in the WinSxS folder. If you get the error and notice the files aren't there, its fairly trivial to troubleshoot and fix.
I'm not a fan of side by side assemblies, I just hate to see issues like this blown out of proportion as it obscures some of the real issues that developers face when developing for Windows (such as just about every bug filed on Microsoft Connect being closed as "by design" instead of being worked on or at the very least closed as "can't fix, compatibility issues", for example).
Re:Speaking as a user (Score:3, Informative)
You seem to be under the mistaken impression that if a library is dynamically linked, then the entire library must be loaded before any program can use a portion of the library. That's not true.
Consider a library with symbols A, B, and C, with programs P and Q. Program P uses symbol A from the library, and program Q uses symbol B from the library.
If these programs are statically linked and are run simultaneously, the memory image of P will be P+A, and the memory image of Q will be Q+B. The code for C will not be loaded because it's not linked into any program.
If the programs are linked dynamically, then the memory image of P is still P+A, and likewise Q is still Q+B. Only the used portion of a library is paged in from disk.
Re:Speaking as a user (Score:1, Informative)
OS X uses frameworks, which can be installed in /Library/Frameworks or ~/Library/Frameworks. If an application wants to ship with its own framework, it will store it in the application bundle and link from there.
Re:And by all developers you mean (Score:3, Informative)
Re:Non-issue for actual msdn coders like myself (Score:3, Informative)
Whether or not that's true, either way this is a non-story. The point is that the OS makers already address this exact situation with technology X, and since he's not using technology X he has a program that no longer runs. That's not Microsoft's fault any more than it would be Linus' fault if his buggy application ran on Linux.
Re:Also... (Score:2, Informative)
Mod parent up.
This is the real world out there described above.
Is the OP not doing something wrong? (Score:3, Informative)
First off, why on earth is the developer still using Visual Studio 2005? We're on Visual Studio 2008, SP1. That, right there, raises a red flag. If someone compiled something with an ancient version of gcc and found out it didn't work, when distributed, on more up to date Linux distributions, wouldn't you think that the appropriate response would be for our man to get his tools straightened out first?
I would think that if the author shipped his system with a copy of the runtimes and had them install in his application directory, he would have no problem at all. The Windows DLL load order is application directory first, then, some other stuff, so his application should always have the right libraries, if he shipped them. In fact, I even think there's some sort of a doohickey that you can do to have Windows look for COM components first in your own directory before it looks for them in common areas. There's no need to have "DLL hell" at all, unless the developer really asks for it.
Frankly, I doubt DLLs of the relatively small size of the CRT should even be used any more across different applications from different vendors.
1. First, you cannot possibly test your application with all the different versions and patch versions of DLLs that are out there, because patch releases now are way too fast. Reliability, right now, not performance, is the pre-eminent problem in the software community.
2. The static linker is now actually capable of removing unused code, from an image, it could not do that before.
3. DLLs have to be relocated when an application loads them into its own process space, so you take a performance hit there.
4. The Windows API has 95% of what you would need the C runtime to do. This isn't like Linux, where, you would die without libc trying to make heads or tales of syscalls and what not. On Windows, I struggle to think of a CRT call that could not be done nearly as simply as in SDK directly. For files there's CreateFile, WriteFile, etc. All of the basic string functions exist within the SDK, and the stuff to display formatted strings in the SDK is better than what the CRT gives you anyway. It's a bit more involved, but, there's articles out there on how to not have a CRT at all. In fact, applications that use the ATL and WTL frameworks even support not having the CRT code, just so you can write really, really tiny applications and gasp, COM components.
Re:Is the OP not doing something wrong? (Score:1, Informative)
VS2008 automatically adds a manifest that requires use of SxS. You don't get around it by simply copying the dlls you want to use into the app directory.
Re:Also... (Score:3, Informative)
Because you can't really expect your customers to upgrade their apps,
No, you can't. Really. Remember, the vast majority of business software, in sales and in number of titles, is proprietary software written in-house or by some consulting dude. A significant portion of this software is mission-critical and replacing it is mostly nor practical (cost) or it is impossible since the original author and often the source code, is long gone/dead etc.
Backwards compatibility is a huge issue for business apps, and Microsoft has been doing a phenomenal job of it. Not that this stops the whiners from whining.
Re:Also... (Score:4, Informative)
>>Microsoft takes backwards compatibility very seriously.
They always say this, and then fairly fundamental programs always seem to break during one of their OS upgrades. Like my very expensive copy of MATLAB. Won't run on Vista. Period. And the code we've licensed won't run on more modern versions of MATLAB. And it's impossible to find high end visualization laptops these days with XP on them.
And yeah, I am a little bitter about it.
Re:And by all developers you mean (Score:5, Informative)
You are 100% correct. The "problem" with those guys is that they did not even understand that they have to redistribute CRT DLL with their binaries even before the update. They got lucky there because on most systems, it is already there (I believe Vista has it out of the box, and probably some MS software will install it on XP - maybe even IE7/8?), so they got away with not distributing it. Now the update brought in a newer version of runtime, which of course the clients don't have, and they suddenly realize that they didn't distribute their dependencies properly. What does "DLL Hell" or side by side assemblies even have to do with it?
And, if they find it so hard to comprehend the idea that they have to give users the DLLs they link against, they could always just compile CRT statically and be done with it.
Re:Discussed on Visual C++ blog (Score:3, Informative)
I agree that updates to IDE & toolchain that introduce new dependencies for binaries produced by that toolchain shouldn't be silent. On the other hand, installing the application on a clean system (for every OS you support, and counting each major Windows version separately) and checking if it runs is one of the most basic tests that every new build of the product should go through. It's trivial to automate, too (if the installer is written properly and allows for silent installation) so there's no excuse not to do it even if you're a 1-man team.
Good news: in Visual C++ 2010, CRT will no longer use SxS [msdn.com] (instead they'll do the same thing Win32 guys are doing and guarantee back-compat with updates, and change the filename if they cannot do that).
Re:Speaking as a user (Score:3, Informative)
Re:Speaking as a user (Score:2, Informative)
Try a real programming language, like C, C++, and other languages with machine-code compilers, not souped-up interpreters (not even souped up in VB's case.)
Delphi does not produce interpreted code and it never has. It has produced native code since Delphi 1 and Turbo Pascal did the same before that.
;))
(Delphi.NET does though.
The most important concept in WinSxS (Score:5, Informative)
Hands up anyone who knows what an "activation context" is! If you don't you have no idea what WinSxS does, how it does it or how to diagnose it when it goes wrong.
In my opinion, WinSxS is a good mechanism, or at least as good as Microsoft could have made it while working within the constraints of history. However, WinSxS cannot be used in the real world without properly understanding it, and achieving that understanding is very painful indeed. The MSDN documentation is piecemeal, appears incomplete and inaccurate in a few places and lacks a proper overview. I think the only reason I properly twigged what activation contexts are about is that I had recently written a mechanism that operated on similar principles (a thread-local stack of context-related information).
I wrote a Wiki page at work describing what (I think) WinSxS's motivation is, how it works and some of the problems it suffers from. I'd like to put it somewhere on the public internet - any suggestions? It should ideally be somewhere wiki-like where people can correct its inevitable inaccuracies without me having to maintain it, but I'm not sure it's appropriate for wikipedia.
Re:Also... (Score:3, Informative)
Who was it that has congratulated the software industry for consistently undoing all the amazing hardware advances of the past decades by responding to bigger memories and faster processors by making the software use more memory and cpu cycles?
That would be all the people for whom software is a tool, not intellectual masturbation. The people who consider "how soon can you deliver", "how much will it cost" and "how long can you maintain it" substantially more important than "will it run on my 10 year old computer".
SxS is a major step forward in this technique, and promises to lead to major increases in memory sales in the near future. And all it took was realizing that a practice that had been considered sloppy wastefulness of memory was actually a valuable new feature for backwards compatibility.
For the vast majority of people, backwards compatibility is far, far more useful than saving a dollar's worth of RAM and CPU cycles.
Re:And by all developers you mean (Score:3, Informative)
That's not a Visual Studio C++ issue, it's the way Windows memory management works. No matter what IDE/compiler/CRT you use, memory allocated by one dll cannot be (reliably) freed by another. It has to be freed by the same dll that allocated it. I'm not 100% sure, but I think you might also run into the same issue if you unload the dll and reload it before freeing the memory.
Re:And by all developers you mean (Score:2, Informative)
You could distribute source on Windows too.
Re:You should not blame Microsoft for this (Score:4, Informative)
I don't think they actually suck. I think they are simply pressurized to produce results in the quickest time possible. I think part of the blame goes to the commissioning clients.
You just haven't worked in enough offices. Believe me, we have people working for our company, producing software, who couldn't code their way out of a wet paper bag. And they're not under any particular time pressure. (We also have a guy who does really good work, but he's so slooow... in that case, you're right, but from my experience that's the minority.)
And part of the blame goes to the "simple is best" mentality in some schools of thought. Simplicity is NOT elegance.
Possibly; but if you know that your shop has coders who won't understand the elegant code, you're better off writing simple code to accomplish the same task. Otherwise, they'll fuck up your elegant code and produce buggy, bloated results. Joel wrote an article about this recently: http://www.joelonsoftware.com/items/2009/09/23.html [joelonsoftware.com] making that very point.
Personally, I don't think elegance should be the goal. Who cares if it's elegant? The finished product is the important part, not the code and the UI is what people judge. I'm sure a lot of products with extremely elegant code have godawful UIs.
Also, opinion is NOT fact.
Shocker!
Surprisingly, I see Cobian Backup, Avast! antivirus, and some other software being rather multi-user aware. Cobian v8 is open source, Avast! Home is freeware. These little attention to details impressed me a lot.
Being multi-user aware isn't "a little detail", it's an industry standard! If they weren't previously for the NT series, they should have been patched to be multi-user away for Windows 2000 Pro, which was mainstream.
That's not "impressive" that's just par for the course. That's like being "impressed" that it uses menus and buttons, or being "impressed" that it lets you switch which language the UI is in.
Then what do they rely on? Their IDE's ability to suggest methods and properties when they type?
That. Or Googling it. Or yelling, "hey Bob, what do you use to sort a list?" over the cubicle top. Or they keep an extensive collection of code snippets and they just copy and paste those in randomly until something works, or appears to work at first glance.
Re:And by all developers you mean (Score:3, Informative)
Indeed. A stanard Windows DLL is not quite the same thing as an assembly, and as far as i know only assemblies have ever been "side by side." If you're doing plain old C++ on Windows, sxs doesn't enter into it, because side by side is a feature of the .net runtime.
Sorry, but you're wrong here. Microsoft is using the term "assembly" to refer to both managed and native DLLs since XP was released (which is before the first version of .NET was even out). Specifically, when you write a manifest file for your binary - which is something that first appeared in XP - you use the <assembly> element as root.
Also, SxS is quite specifically designed to work with native assemblies (for managed, GAC takes care of it).
Here [microsoft.com] is the reference:
Side-by-side (SxS) assemblies allow applications to specify which version of a system DLL or classic COM component to use, such as MDAC, MFS, MSVCRT, or MSXML.
Of course this all still has very little relevance to TFA, as they'd have troubles either way, SxS or not. The only difference is that with SxS they get a clear error report - "This program requires DLL version X.Y.Z" - while without it, they'd just get random crashes (or worse yet, silent data corruption) because of broken ABI between two versions of the binary.