Firefox Too Big To Link On 32-bit Windows 753
An anonymous reader writes "Firefox has gotten so large that it cannot be compiled with PGO on a 32-bit linker anymore, due to the virtual memory limitation of 3 GB. This problem had happened last year with 2 GB, which was worked around by adding a/3GB switch to the Windows build servers. Now the problem is back, and things aren't quite that simple anymore."
This only affects the inbound branch, but from the looks of it new code is no longer being accepted until they can trim things from the build to make it work again. The long term solution is to build the 32-bit binaries on a 64-bit system.
Trying to do too much (Score:5, Insightful)
Re:Wow (Score:4, Insightful)
Some of us actually think that using a web browser is more important than compiling a web browser.
Seriously. My resource usage rarely goes about 1 GB with multiple applications open. These days, the hard drive is a far bigger bottle neck than RAM. Well, unless you're compiling Firefox it appears.
Time to move on, perhaps? (Score:2, Insightful)
No, the long-term solution involves freezing the 32-bit version as an eternal final-state "stable" branch, and moving on to the 64-bit world.
Yes, most people still run 32-bit hardware. And yes, most likely someone would maintain 32-bit backports of FireFox for years to come. But the Mozilla foundation needs to direct their efforts on the reality of the present, not cripple themselves in the interests of backward compatibility.
FWIW, I write this as someone who would primarily use that 32-bit backport at the moment. But I don't expect Mozilla to support my aging XP boxen any more than I expect them to support the Timex Sinclair 1000 or Atari ST I have stashed in the bottom of a closet somewhere.
Last paragraph in the TFA is... confusing (Score:3, Insightful)
"First tests indicate that, for example, moving parts of the WebGL implementation to one side could save 300 KB. In a test run, the newer version of Visual Studio required less memory than the one that was previously used, and 64-bit Windows offers 4 GB of address space."
So, first of all, saving 300KB on WebGL seems like a pittance. Then, there's what appears to be the blatantly incorrect statement of 64-bit windows offering 4GB of address space - shouldn't that be way bigger, or am I stupid?
Re:whose bloat (Score:3, Insightful)
No, no it doesn't, but I'm impressed, turning this against MS instead of complaining about the real culprit, kudos.
Re:The code gets larger, and yet things dissapear! (Score:5, Insightful)
If Seamonkey adopt this silly fade-in fade-out floating toolbar-as-status-bar-replacement, that'll be the end of the Mozilla line for me. Seamonkey has always been the sensible browser for Netscape-heads from way back when since Mozilla Suite died a death and SeaMonkey came into being.
I don't understand the issue (Score:5, Insightful)
It sounds like a build-tool chain problem, not an issue with the eventual binary produced for Firefox.
Why not just run the 64-bit tools on a 64-bit platform and have them output 32-bit code, the same as can be done by virtually any cross-platform compiler system. I can't imagine worrying about the fact that I can't run the builds on an outdated 32-bit OS as long as I can produce the binaries for such platforms.
I shudder to think what it would have been like to develop for some of the military communications systems I worked on for my first job if we had to run the compilers on that pathetically slow mil-spec Sperry AN-UYK system (magnetic core memory -- talk about slowing down the CPU! But it's radiation hard!) We only tested on the Sperry -- all builds and editing were done on a VAX.
In modern terms, could you imagine having to run your editors and compilers on an iDevice instead of OS/X?
Re:Well done (Score:2, Insightful)
There's something seriously wrong when a browser hits a 3GB linker limit.
Imho, the only sane solution is to delete most of the code. It's been crap for years anyway.
Did you ever try compiling Chromium before you made that statement?
Re:VS 2005? (Score:5, Insightful)
Seems ironic that the FF team is using stuff from seven years and two major versions ago while at the same time bemoaning that anybody might want to keep a version of Firefox for more then 6 weeks - especially enterprise users.
Interesting how they don't practice what they preach.
Old timer chimes in (Score:5, Insightful)
Back in my day we didn't have gigabyte memory and disk space was at a premium.
It might seem a bit strange now, but back in the good 'ol days we used to have to break up a project into separate components, just in order to compile it!
This is where your interface and API design skills came in handy. If you could partition some piece of the project off into it's own DLL, you could effectively break up the project into smaller pieces that could be individually compiled.
That's where the name DLL came from originally: "Dynamic link library". You didn't need to have all the code read into memory when you first executed the application - less commonly used features wouldn't get loaded until they were actually asked for.
It's not like it is nowadays, where you actually need all the code to be available all the time. "Rich user experience" they call it.
I suppose it's just the future overtakin' us. Them good old days is gone forever.
Re:Oh, now we admit it is getting bloated (Score:5, Insightful)
I'd say it nicer but then I'd risk not hitting +5 Funny.
Re:Trying to do too much (Score:5, Insightful)
I think it's still indicative of the problem GP mentions. The more code you are trying to pull in, the larger the footprint during the build process. You don't see a 'Hello world' program requiring a 3GB+ build footprint do you? No, because it's not doing enough to warrant that. Likewise, Firefox apparently *is* trying to do a lot. More than it used to at any rate.
Re:Wow (Score:5, Insightful)
No, it has nothing to do with running Firefox. It has everything to do with running Visual Studio's linker.
This matters only to Firefox developers.
Not that they shouldn't care, mind you, as that is some seriously monolithic code. But it won't make any difference to Joe Sixpack.
Re:whose bloat (Score:3, Insightful)
Templates out the wazoo. Don't blame me, I didn't design it. (Or write most of it; I'm only tangentially related nowadays.)
Point being, there are features which, while valuable, are costly. I blame our source, our fondness of templates, and the compilation consequences that templates almost necessitate way more than I blame GCC for having a poor implementation of templates.
But similarly, going "stupid bloaty MSVC, it shouldn't support this feature which can give double-digit percentage speedup because it takes a lot of memory" is stupid. LTO is costly -- period. It's just very worth it.
Re:Trying to do too much (Score:5, Insightful)
Thank you for posting this. Filling the virtual address space of the linker probably does indicate some problems with the Firefox source code - crazy big translation units, for example - but it doesn't imply anything about the size or quality of the resulting binary.
I thought people on this site were supposed to know something about computers.
Re:Time to move on, perhaps? (Score:2, Insightful)
No, the long-term solution involves freezing the 32-bit version as an eternal final-state "stable" branch, and moving on to the 64-bit world.
Hahaha! Uh, no.
1. The MSVC compiler is a 32-bit program. Sure, it can compile 64-bit code, but it is itself 32-bit, so using a 64-bit OS doesn't even help you. cl.exe is a 32-bit process. Always.
2. All of Windows is compiled with that compiler. ALL OF WINDOWS. That means there is NOTHING in Windows that requires this much VM in order to build. So Firefox is more bloated than ANY component within Windows.
Seriously, anybody with the slightest clue about how software works should realize that this is the symptom of some incredibly deep-rooted problems within the architecture of Firefox.
Re:Wow (Score:5, Insightful)
Re:Trying to do too much (Score:2, Insightful)
It's more an indication that compiler writers are taking advantage of the huge amount of memory available for optimization techniques that take a lot of RAM. This is actually a GOOD thing. The bad thing is if it takes RAM while it's running.
Re:No PAE?! (Score:3, Insightful)
You mean some people still run a 32-bit OS?
Not only that, but apparently Windows cannot use PAE - Physical Address Extension [wikipedia.org] to address more than 4GB (according to the WP entry, PAE is supported, but the 4GB limit is still enforced - due to some obscure licensing problems).
The problem is with virtual memory. A process still uses 32bit memory addresses to reference memory. This means that a process can still only address 4GB of memory. If you were to use more than 32bit memory addresses to get more memory, suddenly you aren't a 32-bit OS anymore.
PAE only helps the OS be able to manage more than 4GB of memory.
Re:Trying to do too much (Score:5, Insightful)
The "translation unit" involved here is the whole binary. We're talking about link-time code generation with profile-guided optimization, not regular compiles.
So it doesn't indicate much of anything about the Firefox source code other than general overall quantity of code being compiled...
Too funny reading these comments (Score:5, Insightful)
Folks are crawling all over each other to show how ignorant they are. "I ditched firefox for Chrome cause it's lighter!" only to ignore the fact that Chrome also has the same problem with the PGO thing running out of RAM so they don't even bother with trying those optimizations anymore. "Geez Firefox needs more RAM than the kernel to compile? Something's wrong!" Yes if the Linux kernel was built with PGO on VS 32-bit it probably would run out of RAM there too. Then there's the guy that claims this PGO problem is evidence that the Firefox devs need to go back to remedial school. I'm sure he could do it far better (and avoid the PGO linking optimization running out of memory too!).
Hilarious reading. At least I choose to laugh rather than cry at people's inability to read and understand the issues here.
Re:Eg? (Score:5, Insightful)
> 1) What the hell are you doing with your code to be
> that large?
How large? It's a few million lines of C++ code, just like every other browser. What it's doing is implementing all the stuff people want to do on the web.
> 2) What the fuck is your linker doing to do that?
Please read up on link-time code generation.
> 3) Why the hell didn't you see this coming and
> prune LONG before you hit the 3Gb limit if you
> already hit the 2Gb limit once already?
This is an excellent question that I too am asking.
> 4) What's the problem with compiling on 64-bit
> computers only,
None, except updating a large build farm from 32-bit to 64-bit can't happen overnight. Needs some staging, testing, etc.
> You're honestly telling me that Firefox is more
> complicated and needs more memory to compile
> than, say, LibreOffice?
Have you tried to compile LibreOffice with LTO and PGO turned on?
> The Linux kernel?
Quite possibly. The kernel is C, not C++; C++ is a lot more of a pain for compilers to deal with.
The whole point of LTO is that you optimize your entire binary as a single object, no matter how your code is structured. It requires more memory, but can produce faster code because the compiler is able to make optimizations it can't make otherwise.
Whether that's "crappy" is a matter of your priorities, of course. 10-25% performance improvement tends to be a high priority for web browsers, though perhaps not for KDE or LibreOffice.
Re:Wow (Score:2, Insightful)
Re:Time to move on, perhaps? (Score:4, Insightful)
Possibly not. But part of the reason is that it is not possible [microsoft.com] to set 64-bit IE9 as the default browser on Windows Vista or 7.
That, and no official releases of 64bit Firefox, Chrome, (Safari?) or Opera for Windows. So it's really a team effort to keep 64bit browsing off Windows.
Re:Time to move on, perhaps? (Score:4, Insightful)
If you say so, but if MS used PGO for things like MS Office, IE, or even the windows explorer shell I'm sure they'd quickly run out of RAM as well.
So no, Firefox is not more bloated than "*ANY* component within Windows."
Seriously, anyone with the "slightest clue" about what the article is talking about would understand that this issue is not about "deep-rooted problems" in Firefox. And as the other poster mentions, MS knows about this problem and now has 64-bit binaries of the compiler and linker now so this is less a problem. However it only addresses the issue for 64-bit apps; the 64-bit binaries don't appear capable of compiling and linking 32-bit apps, if I read this correctly.
Re:No PAE?! (Score:1, Insightful)
Windows most certainly has no problem using PAE, I've been using it for 10+ years. The idiot quoting Wikipedia didn't bother to read the next sentence which refers to Address Windowing Extensions ... which all 32 bit apps to access more than 4gigs of ram on Windows. Other OSes are no different, if you think they are its because you don't understand PAE and how it works.
But more physical address space doesn't help here, the problem here is virtual address space for running an effective but memory hungry profile guided whole program optimisation process. Nromally 32-bit windows has a maximum of 2GB virtual memory per process (and this is one big process we are dealing with). This can be increased to 3GB at the cost of reducing the kernel address space to 1GB.
You do not understand what PAE is.
PAE is an effective way to access far more than 4 GB of memory on a 32 bit system. Its another form of paging (NOT SWAPPING). Its essentially FAR pointers from the old days.
Going to a 64-bit OS (which allows 4GB of virtual address space for 32-bit processes)
No, it doesn't. The 32 bit processes are still constrained as they were before as the kernel still has to present itself to the application by being in the same place it was on a true 32 bit machine. 32 bit apps can't magically address more than 4GB of memory just because the OS can, so the kernel still presents itself in that 4GB, which means ... you still get at best 3GB of usable space per application, even though you can have many more applications all accessing ram that totals more than 4GB.
What they really need to do is address the fact that their WEB BROWSER CAN NOT BE COMPILED IN 3 FUCKING GIGS OF RAM.
Let me repeat that for you.
A WEB BROWSER THAT CAN NOT BE COMPILED IN 3 FUCKING GIGS OF RAM.
You can compile the ENTIRE OS without a problem ... but a web browser ... not so much ...
And yet Mozilla and the other Firefox fanboys still fail to see the writing on the wall.
Re:Trying to do too much (Score:4, Insightful)
Just because Chrome is bloated too doesn't mean Firefox isn't.
Re:Wow (Score:5, Insightful)
Not that they shouldn't care, mind you, as that is some seriously monolithic code.
They're talking about using link-time optimisation (LTO). That means that you compile each compilation unit to an intermediate representation and then run optimisations on the whole program. This takes a staggering amount of memory (which is why no one bothered with it off high-end workstations with very expensive compilers until very recently), but can sometimes be worth it. It actually helps modularity, because you can keep your source code nicely separated into independent components without worrying about efficiency, and you can do better data hiding.
As a trivial example, consider an accessor method. Something like getFoo() { return foo; }. Without LTO, you'd want to put the declaration of the class and this method in the header so that every compilation unit could separately inline it. This reduces modularity, but it saves you the cost of a function call just to access a field in a structure. With LTO, you can make the class opaque (if you're a C++ junkie, using the Pimpl pattern, if you're a C programmer by just not declaring the struct in the header). You'll get a single copy of the function emitted, and then the link-time optimiser will inline it everywhere.