Hardware Virtualization Slower Than Software? 197
Jim Buzbee writes "Those you keeping up with the latest virtualization techniques being offered by both Intel and AMD will be interested in a new white paper by VMWare that comes to the surprising conclusion that hardware-assisted x86 virtualization oftentimes fails to outperform software-assisted virtualization. My reading of the paper says that this counterintuitive result is often due to the fact that hardware-assisted virtualization relies on expensive traps to catch privileged instructions while software-assisted virtualization uses inexpensive software substitutions. One example given is compilation of a Linux kernel under a virtualized Linux OS. Native wall-clock time: 265 seconds. Software-assisted virtualization: 393 seconds. Hardware-assisted virtualization: 484 seconds. Ouch. It sounds to me like a hybrid approach may be the best answer to the virtualization problem.
"
Sponsored by VMWare.. what do you expect? (Score:5, Insightful)
Re:Sponsored by VMWare.. what do you expect? (Score:3, Insightful)
Even so, they may be at least partially right.
Besides, if a hybrid approach is necessary, VMWare will need to adjust as well. Or am I missing something?
Re:Sponsored by VMWare.. what do you expect? (Score:5, Informative)
But Vmware's agitation is understandable. They're about to lose it all to an open source project. Where have I seen this before?
Re:Sponsored by VMWare.. what do you expect? (Score:5, Informative)
Note that Xen's original hypervisor implementation *is* a software solution -- it relies on rewriting the guest operating system kernel so that the kind of hardware traps that VMware are talking about here are unnecessary. Note that it worked flawlessly before the virtualisation technology (eg. Intel VT) that VMware is testing was avialable.
This HAS happened before - with Stacker (Score:3, Informative)
This won't be the first time software beats hardware.
The original Stacker product was a combination of a hardware card and software. Think of the hardware card as an accelerator for doing the comression/decompression.
The hardware was faster on the oldest machines, but on anything above a 286/12 (I had a 286/20 at the time), or almost any 386, it ran faster without the hardware card. And on every 486, the card was useless.
So, while you may want to "consider the source" of this news, this is only one f
Re:This HAS happened before - with Stacker (Score:2)
The Stacker thing is simply because the other hardware
Re:This HAS happened before - with Stacker (Score:2)
"I wonder if anyone still uses Stacker?"
Why bother ... you can get a a terrabyte of storage for less than the cost of a 40 meg hard drave back in those days.
Re:This HAS happened before - with Stacker (Score:2)
One good example of that is the "elevator" algorithm that linux uses to write to disks. Instead of moving the head from place to place in the order writes were supposed to be written, it "sweeps" back and forth across the surface of the disk. A lot less wear and tear on the drives, and it also allows you to continue to use a bad drive way after it won't work under a certain other OS.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Citrix?
Not an open source product and not lost it to an open source product, but they made a product that has been largely made superflouos because MS built it right into the OS.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
http://en.wikipedia.org/wiki/Terminal_Services#Ci
Re:Sponsored by VMWare.. what do you expect? (Score:2, Interesting)
I'll let you in on a secret: if you consider all costs, and return on investment, using VMware is a competitive advantage over using Xen.
Say whaa??? (Score:2)
Xen: free. Linux: free. I don't understand where I would spend any money turning 1 linux server into 2 linux servers ith Xen. We don't use Windows on anything but the domain controllers, but Xen doesn't windows (nor would I want to virtualize our DCs...)
Re:Say whaa??? (Score:2, Insightful)
Re:Say whaa??? (Score:2)
Re:Say whaa??? (Score:2)
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Reports of VMWares demise have been greatly exaggerated. They're just reacting to a new threat. VMWare is an EMC company, and I doubt EMC is going to let virtualization die. The future of EMC's 23.9 billion dollar empire depends on their ability to virtualize and cluster their machines. This is the quiet before the storm in the storage market...
Re:Sponsored by VMWare.. what do you expect? (Score:2)
--- Um, I dunno, where? Oh, you mean how open source Linux and BSD have killed off proprietary OS's like Windows. Yeah ... I've seen it before...or are you referring to something else? :-)
Excuse my ignorance, but last I heard [dated info], was that Xen re
Re:Sponsored by VMWare.. what do you expect? (Score:3, Insightful)
It's like Apple's claim that their Intel jobbies are 5x faster - a bit silly and very, very specific...
And yes, VMWare are hardly likely to mention that Xen-style virtualisation is going to be better now, are they?
Re:Sponsored by VMWare.. what do you expect? (Score:5, Insightful)
VMware's paper is a typical research paper, published at a peer-reviewed conference. This means that they have used the scientific method. The chances are 99.9999% that you will easily reproduce their results, even if changing the benchmarks.
I, on the other hand, am smart enough to see that they are stating the obvious. If you read the Intel VT spec, you'll see that Intel does nothing for page table virtualization, nor anything for device virtualization. Both are extremely expensive, and besides sti/cli, are the prime candidates for hardware assists. Intel will likely solve this performance issue in future revs, but right now, VT isn't fast enough.
Hmmm, virtualisation? Do you happen to work on Xen?
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Re:Sponsored by VMWare.. what do you expect? (Score:3, Insightful)
And by the way... yes... device virtualization is still not there, but your page tables claim is bullshit. If you read the VT (and the SVM) docs, you would realize that you can implement shadow page tables RIGHT NOW. The hardware assists are there.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Re:Sponsored by VMWare.. what do you expect? (Score:5, Informative)
Disclaimer: I work for VMware.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Yes, AMD Pacifica seems to be far better (Score:4, Interesting)
Apparently, yes, and by a good margin.
There are several documents and articles out there which point out VT's problems and how Pacifica is quite dramatically better. Here's an excerpt from "AMD Pacifica turns the nested tables" [theinq.net], part 3 of an informative series of articles:
This should allow an otherwise identical VMM to do more things in hardware and have lower overhead than VT. AMD appears to have used the added capability wisely, giving them a faster and as far as memory goes, more secure virtualisation platform."
So, it looks like AMD are ahead on hardware virtualization at the moment.
If I read it correctly, this is because Intel's VT actually requires a lot of software intervention, so it's not actually a very strong hardware solution at all.
Re:Yes, AMD Pacifica seems to be far better (Score:2)
Re:Yes, AMD Pacifica seems to be far better (Score:2)
Pacifica has a slight advantage in that it supports ASIDs (Address Space IDs, see your OS textbook's section on page tables), a long-overdue x86 feature. But even theoretically, that's not going to make up the difference.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
But hey, let's hear it for correctness!
I designed h/w virtualization (Score:3, Interesting)
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Actually the reason why VMware will use a slower approach is becouse of Xen.
If there's any condition were HW virtualization is slower than SW virtualization, you need to use it or lose to Xen Source.
Don't get me wrong, I work with VMware in Argentina and the guys that sells VMware here are really great to work with. I love the technology and the easy it is to manage it. But certeanly Xen+HW virtualization changes everything....
The worst thing that VMware has is the the EULA p
Re:Sponsored by VMWare.. what do you expect? (Score:5, Informative)
As far as the results there is nothing surprising here. This has happened before. Fault driven emulation of 80287 was nearly 50%+ slower than compiled in emulation. There were quite a few other examples x86 which all revolve around the fact that the x86 fault handling in protected mode is hideously slow. Last time I have had a look at it in asm was in the 386 days and the numbers were in the 300 clock cycle range for most faults (assuming no wait on memory accesses). While 486 and Pentium improved the things a bit in a few places, the overall order remains the same (or even worse, due to memory waits). Anything that relies on faults in x86 is bound to be hideously slow.
Not that this matters, as none of the VM technologies is particularly caring about resources. They are deployed because there is an excess resource in the first place.
Re:Sponsored by VMWare.. what do you expect? (Score:2)
Measuring the wrong stuff (Score:2)
If hardware solutions can do a better job of compressing the memory that's not in use (unlikely) or virtualising 3D video, so tha
Don't talk if you don't actually know (Score:2)
So don't be
Nevermind who sponsored the study... (Score:2)
Re:Nevermind who sponsored the study... (Score:2)
Hybrid? Good + Bad = Better? (Score:2, Insightful)
* - I imagine in real life it's not a 1:1 ratio, but for the sake of argument, work with me.
Re:Hybrid? Good + Bad = Better? (Score:3, Insightful)
I suppose there are certain things hardware virtualisation does better.
The trick is, I'd guess, to find out which works better in which circumstances.
You see that people suspect this white paper because of its origin; they are right in doing so at least because only one type of test has been performed; surely not all computing tasks perform the same way as a kernel compile.
This suggests that VMWare have found the example which supports their claims the best; the question is, of course, whether this is th
Re:Hybrid? Good + Bad = Better? (Score:2)
CAPTCHA: pitying, how appropriate.
Re:Hybrid? Good + Bad = Better? (Score:2)
For example, say you have a boat powered by 393 horsepower engine and a 484 horsepower engine. If you run them both at the same time, the net power is not going to be 439hp.
Software+hardware won't add in nearly the same way, but I wouldn't be surprised if a hybrid approach was %50 faster than either method alone.
Re:Hybrid? Good + Bad = Better? (Score:3, Insightful)
The correct conclusion is more limited (Score:5, Insightful)
The correct conclusion is not that virtualization is better done entirely in software, but that current hardware assists to virtualization are badly designed. As the complete article points out, the hardware features need to be designed to support the software - not in isolation.
It reminds me of an influential paper in the RISC/CISC debate, about 20 years ago. Somebody wrote a C compiler for the VAX that output only a RISC-like subset of the VAX instruction set. The generated code ran faster than the output of the standard VAX compiler, which used the whole (CISC) VAX instruction set. The naive conclusion was that complex instructions are useless. The correct conclusion was that the original VAX compiler was a pile of manure.
The similarity of the two situations is that it's a mistake to draw a general conclusion about the relative merits of two technologies, based on just one example of each. You have to consider the quality of the implementations - how the technology has been used.
Re:The correct conclusion is more limited (Score:2)
Re:The correct conclusion is more limited (Score:2)
Re:The correct conclusion is more limited (Score:2)
Linux can run on the bare metal (the only OS on the entire system), as a first-level image in an LPAR (the LPAR is actually managed by a lightweight hypervisor), and on top of of z/VM (itself on top of the bare metal or an LPAR) which is
Re:The correct conclusion is more limited (Score:5, Interesting)
It also had a few other advantages. Since you were adding virtual instructions, they all completed atomically (you can't pre-empt a process in the middle of an instruction). This meant you could put things like thread locking instructions in the PALCode and not require any intervention from the OS to run them. The VMS PALCode, for example, had a series of instructions for appending numbers to queues. These could be used to implement very fast message passing between threads (process some data, store it somewhere, then atomically write the address to the end of a queue) with no need to perform a system call (which meant no saving and loading of the CPU state, just jumping cheaply into a mode that could access a few more registers).
Re:The correct conclusion is more limited (Score:2)
I'm confused, why would you need a system command to pass messages between threads? Isn't that what atomic read-and-write ASM commands are for? That plus thread-shared memory
Re:The correct conclusion is more limited (Score:3, Funny)
Their entry models (10k US$) are slow as shit though. Can't say anything about the more expensive machine, but anything that requires around 12 hours to upgrade it's operating system can't be trusted.
Re:The correct conclusion is more limited (Score:3, Interesting)
Note that the 'naive conclusion' and the 'correct conclusion' are not contradictory: I remember an article recently where it was shown that the Alpha had three times the power of a correspondig VAX, which made nicely the point that CISC is shit.
Now as Intel has shown, given enough efforts and money even x86 the poorest CISC ISA ever (VAX ISA was much nicer than x
Re:The correct conclusion is more limited (Score:2)
This was heavily discussed a while ago on comp.arch. Conclusion: VAX instruction set was an absolute nightmare for hardware designers; while today the problem of making x86 fast in spite of the instruction set is basically solved, making a VAX fast would have taken superhuman effor
Re:The correct conclusion is more limited (Score:2)
Sure some VAX instruction such as 'list management' cannot really be made fast, but the x86 has also such kind of instructions, but those instructions are irrelevant, they can be trapped and handled by microcode, and the compiler writers avoid those instruction as they know that they are slower than doing it 'by hand'.
I would have thought the 16(if memory serves) orthogonal registers would have made a nice target for compilers, contrary to the ridiculous n
Re:The correct conclusion is more limited (Score:2, Insightful)
I read that the calling convention specified a general call instruction which was architected to do a lot of stuff - build a stack frame, push registers and so on, so even an efficient implementation will be slow. Much of the time, you could get away with something much simpler.
>I would have thought the 16(if memory serves) orthogonal registers would have made a nice
>target for compilers, contrary to the ridiculous number of (non-orthogonal) reg
Re:The correct conclusion is more limited (Score:2)
All that limiting instructions to gain performance means is that the inst
I smell a straw man... (Score:2, Interesting)
Perhaps the intended conclusion was that it was feasible to write an efficient compiler using only a small, intelligently chosen with compiler optimization in mind, subset of the instruction set. Perhaps the fact that the original compiler was (as you assert) "a pile of manure" was not unconnected to the fact that it tried to achieve speed by exploiting the entir
Re:I smell a straw man... (Score:2)
The "core" of modern x86 processor is nothing like RISC processors from back when the arguments raged. All modern processors implement designs similar to x86 rather than execute their instruction sets directly. Claiming that these internal engines are "RISC" is preposterous.
The "IS" in RISC and CISC stands for "Instruction Set" an
hardware v/s software (Score:3, Insightful)
Sure, the instructions may be hardcoded, coming out of ROM, or whatever, but in the end its instrructions that tell the hardware what to do. And those instructions are called "software", no matter how the vendor tries to spin it. And if the solutions performs badly, it is because the software is designed badly. Period.
Re:hardware v/s software (Score:2)
No not really (Score:3, Informative)
Re:No not really (Score:2)
Imagine a man with a computer and a man with a pen and paper both tasked
Re:No not really (Score:4, Interesting)
I am not willing, based on a single datapoint, to make any conclusions. That's tanget to my point anyhow, my point was that doing something in hardware and software are quite different.
Re:No not really (Score:3, Funny)
Re:No not really (Score:2)
Presumably because Intel did a lousy job on the implementation. It's not like it hasn't happened before (see: x86 ISA, SSE, memory buses for multiprocessors, etc). Hopefully they will do better for the next version; if not there is always AMD's implementation.
"It sounds to me like a hybrid approach may be th" (Score:2)
As so many times and so many cases before has it proven to be the optimal solution. What gives ? Good is that we have all these alternatives, and every vm company will try to evaluate, then optimize, which will lead to better performing software VMs, and because hw is slower to catch up, probably software VMs will be better for a while.
wrong (Score:4, Insightful)
It may or may not be faster eventually, but that doesn't matter. What matters is that small changes in the hardware make it possible to stop having to depend on costly, proprietary, and complex software--like that sold by VMware.
Re:wrong (Score:2)
Not to say that you're wrong or that hardware should be free-as-in-freedom, but the irony was to great to resist.
Not just the CPU (Score:5, Interesting)
I am 100% in favor of cheap and open solutions. But I don't agree that this will soon be the case for virtualization. VMWare and the few other major vendors do a lot more than software virtualization of a CPU (which is all TFA was talking about). To have a complete virtualization solution, you need to also virtualize the rest of the hardware: storage, graphics, input/output, etc. In particular graphics is a serious issue (attaining hardware acceleration in a virtual environment safely), which from last I heard VMWare were working hard on.
Furthermore, Virtualization complements well with software that can migrate VMs (based on load or failure), and so forth. So, even if hardware CPU virtualization is to be desired - I agree with you on that - that won't suddenly make virtualization as a whole a simple task.
Re:Not just the CPU (Score:2)
Actually, the people who have made the most headway are Microsoft. The Vista driver model is designed for support for virtualisation in mind. This means that the OS has access to video driver commands for things like saving and restoring GPU state. As far as I know, other operating systems currently lack this; Linux has a problem even switchi
Re:Not just the CPU (Score:2)
Actually, Microsoft is just trying to catch up. Linux already has a very flexible driver model in place, and its GUI uses an architecture that is ideally suited to virtualization.
Linux has a problem even switching between virtual consoles
Linux has no problem switching between virtual consoles if you use the correct drivers for your hardware.
really (Score:2)
VMware's big thing was a JIT-like x86 engine, a complex piece of software that is now not needed anymore. That really is a big deal.
Re:Not just the CPU (Score:2)
Kripkenstein mentions the dirty little secret that Intel doesnt tell you. Virtualization of the CPU is just that, CPU Virtualization, the memory management, peripherals, and IO hardware is still managed mostly by a host OS.
So, Software is faster until the CPU vendor includes a bios and chipset thats more virtualized oriented.
Vmware's main advantage is they provide the host
Re:Not just the CPU (Score:2)
Yes, and the reason it does is because the old x86 had a few non-virtualizable instructions; an efficient workaround was a lot of effort, and Xen chose the simpler route of simply disallowing those instructions in the guest OS.
Re:wrong (Score:2)
It may or may not be faster eventually, but that doesn't matter. What matters is that small changes in the hardware make it possible to stop having to depend on costly, proprietary, and complex software--like that sold by VMware.
Maybe I'm crazy, but I just don't see that happening anytime soon in the mainstream. When the
Use Paravirtualization (Score:4, Insightful)
g
Re:Use Paravirtualization (Score:2)
The whole point of virtualization is so you can run your favoriate OS most of the time, and only switch over to Windows when you want to run games, isn't it?
Well, I suppose if choose to stay in the Windows world most of the time, the whole point of VM is to try to keep malware off your computer... But either way, you're not getting a FLOSS paravirtualized Windows kernel any time soon.
Re: Use Paravirtualization (Score:2)
Your windows desktop is not the whole point.
g
Re: Use Paravirtualization (Score:3, Insightful)
Re: Use Paravirtualization (Score:2)
Very true, but a lot of the folks that have been deriding VMware as being inferior to Xen are failing to notice that Linux server farming is not the whole point either. At work, I'm limited to using Windows on my desktop, but I still have quite a bit of Linuxy stuff to do and don't have the space or budget to set up additional machines. I also need to have a flexible networking environment in which to test, so I run VMware Workstation. VMware has proven to
Look to IBM (Score:3, Informative)
Re:Look to IBM (Score:4, Interesting)
Just like what is happening now, they added specific support to the hardware to make VM perform better.
This all happened before the development of today's architectures, but in the early days of microcomputing, IBM had the position that Microsoft has today: they were the big company that had 90% of the market, and in the eyes of the newcomers all they did was by definition the wrong thing. So nobody would bother to look at 360 mainframes, VM and how it was done before designing their own processor.
(this would be similar to telling a Linux geek to look at how certain problems are solved in Windows... it is Windows, it is Microsoft, so it has to be the wrong solution)
Re:Look to IBM (Score:2)
VM370 was a dog. Why? Because they relied on hardware traps and software simulation of CCW's (channel command words), to run the host operating system "perfectly."
A hack to this, used by National CSS and other timesharing vendors (because, remember that CP/CMS was open source software and VM370 was just one implementation of it), was to replace CCW's inside CMS with specific traps for OS services. The result was that National
I think that's a little innacurate (Score:4, Insightful)
It's no supprise that large, extremely expensive computers get technology before home computers do. You give me $20 million to build something with, I can make it do a lot. You give me $2000, it's going to have to be scaled way back, even with economies of scale.
You see the same thing with 3D graphics. Most, perhaps even all, the features that come to 3D cards were done on high end visualizaiton systems first. It's not that the 3D companies didn't think of them, it's that they couldn't do it. The orignal Voodoo card wasn't amazing in that it did 3D, it was much more limited than other thigns on the market. It was amazing in that it did it at a price you could afford for a home system. 3dfx would have loved to have a hardware T&L engine, AA features, procedural textures, etc, there just wasn't the silicon budget for it. It's only with more developments that this kind of thing has become feasable.
So I really doubt Intel didn't do something like VT because they thought IBM was wrong on the 360, I think rather they didn't do it because it wasn't feasable or marketable on desktop chips.
Re:Look to IBM (Score:3, Interesting)
And then there's paravirtualization (Score:2, Interesting)
I think that any kind of "full virtualization" is going to be subject to these issues. If you want to see performance improvements then you should modify the guest os.
VMware's BT approach is very effective and their emulated hardware and bios are efficient, but that won't match the performance of a modified OS that KNOWS it's virtualized and cooperates with the hypervisor rather than getting 'faked out' by some
AMD Engineer told me the same thing (Score:2)
Re:AMD Engineer told me the same thing (Score:2)
Parallels on Mac OS? (Score:3, Interesting)
Parallels on OS X switches between software and hardware virtualization and using hardware virtualization its about 97% the speed all around of native hardware (consider that virtualization on current Yonah CPUs is equal to one core only). Software virt on Parallels is much slower - on par with running Windows Virtual PC on the same box using Windows XP (not Mac Virtual PC).
Combination HW/SW better yet (Score:2)
Random Usage (Score:2)
Doesn't anyone have a clue? (Score:2)
Re:Bias? (Score:4, Insightful)
Re:This is why hypervisors rule (Score:2, Insightful)
I should be sleeping.
Rich
Re:This is why hypervisors rule (Score:2)
Re:best platform anyway? (Score:2)
That would be a cool project; the problem is that the twenty gazillion computers already out there in the world still run the bad old Intel ISA, and most likely the next twenty gazillion computer will too (because economies of scale have made Intel PCs so cheap). That means that 99% of existing computers can run a virtualized Intel ISA at (close to) full sp
Re:best platform anyway? (Score:2)
Re:dual boot (Score:2)
Re:How not to write an isPrime function (Score:2)
Oh, and Slashcode
Re:How not to write an isPrime function (Score:2)
Re:They all seem to be slow pigs (Score:2)
Now run "top". Note how 98% of the time the CPU is twiddling its electronic thumbs in an idle state.
Now, granted, with any half-sane Unix you don't need to set up a separate VM for all the above tasks, but there's a lot of benefits to doing so:
1. A security hole in one s
Re:They all seem to be slow pigs (Score:2)
We routinely use this to build and configure new servers. Pick the least loaded multi-server box, install a new jail on it, and treat it like a new machine. If we ever need to move the system to its own dedicated hardware, we can use tar to move its directory onto the new drive.
Re:They all seem to be slow pigs (Score:2)
BSD jails don't allow a whole different operating system in the VM. If the company needs some fancy commercial product which is "only supported on RHEL 4.(mumble) with THESE patches (but not THOSE patches), yet everything else in the company runs FreeBSD, the argument of "minus the kernel" holds no water whatsoever - as soon as you say "minus the kernel" you're giving the commercial support folks a "get out of jail free" card for any suppor