Going Deep Inside Vista's Kernel Architecture 478
bariswheel wrote to mention an episode of 'Going Deep' on Channel 9 which takes a hard look at the architecture of Windows Vista. From the post: "Rob Short is the corporate vice president in charge of the team that architects the foundation of Windows Vista. This is a fascinating conversation with the kernel architecture team. It's our Christmas present to all of the Niners out there who've stuck with us day after day. This is a very candid interview." Topics discussed include the history of the Windows Registry, and the security/reliability of Microsoft's upcoming operating system.
Re:Is that a word? (Score:2, Informative)
Direct link to video (Score:5, Informative)
That's It?? (Score:5, Informative)
So they discovered software dependencies and configuration management, error handling in the kernel, and reversed one of their previous errors - putting device drivers inside the kernel.
I'm no OS guru (I'm just an applications guy), but shouldn't they have thrown the whole mess in the garbage and started over? They're referring to the Vista kernel as "NT"!! It's freakin NT!
NT's karma has waned (especially this week). God help us - we'll be stuck with MS security holes forever.
Re:"architects" (Score:3, Informative)
Re:I love the questions they ask. (Score:5, Informative)
But of course, backups are automatically made on successful bootups to minimize the damage done if you'd suffer from a file corruption in that specific file. But I've never figured out when it does that. It clearly doesn't seem like on every successful boot, as I've seen messages like "Windows has restored a registry backup" and after that wondered where all settings the past few months went, and why some programs don't even run anymore. Gah... Thankfully last time it happened were a number of years ago. *knocks wood*
Interestingly, Microsoft has started opting more for
MMS stream hails from microsoft.com!?! (Score:2, Informative)
Not to diss the underlying interview [I'm always willing to hear about kernel stuff], but it's kinda odd that the MMS stream originates at a M$FT server: [Slashcode tends to put hard breakline characters and other weird white space into web addresses, so you will probably have to paste that address into a word processor and clean it up].
Re:Please, kill the registry... (Score:5, Informative)
It's hardly a lock-in method when it's both documented methods and it's easy to find out what happens -- the Windows registry is hardly rocket science, but more like a tree of settings that can have a few different data types.
Vista and WMF Vulnerability (Score:5, Informative)
Torrent (Score:3, Informative)
I just posted the torrent, enjoy:
http://64.226.48.88/kernel_windows_vista_2005.wmv
Re:I love the questions they ask. (Score:3, Informative)
C:\Documents and Settings\\Application Data\
(can be sync'ed with a domain server)
and
C:\Documents and Settings\\Local Settings\Application Data\
(remains on this machine only)
Re:I love the questions they ask. (Score:3, Informative)
The registry first existed for registering OLE document types in about Windows 3.0. At the time, it had almost no structure and responsibility. The first release of Windows NT (NT 3.1) made the registry much more important: everything that used to be stored in
Meanwhile, the Windows 9x/ME series still uses
Personally, I don't see what the big problems with the registry are. The registry is a hierarchial database system provided by the OS designed to store configuration information. The only differences between it and a
- The storage is managed by another database on top of the filesystem, instead of the filesystem db directly.
- The seperation of config entries is handled by the db in the registry, whereas each app comes up with its own format in
/etc.
The registry has several documented functions for hot backup and restoration, and has always been journalled (like the fs metadata). A lot of apps abuse the registry, but I think they'd do the same thing with their config files on another OS.Re:I love the questions they ask. (Score:3, Informative)
Interestingly, Microsoft has started opting more for .config XML files stored in the application directory (sort of like their old .ini files) in their new wave of .NET applications, and that seems to be more like the recommended way of storing application settings. I don't know how user-specific settings are dealt with if doing it that way though, and if it's only suitable for settings for the local machine.
There's a special directory for storing user-specific settings. On a default install of Windows XP, it's located at "C:\Documents and Settings\[user name]\Local Settings\Application Data\[company name]\[program name]"
AFAIK, there's no guidelines on what to do if two companies share the same name and the same product, though I guess that would be relatively unlikely.
Transcript (Just Intros - Working On The Rest) (Score:5, Informative)
Narrator: Alright, so we're here for "Going Deep." We have the corporate vice president and some of his architects and they're going to talk about the Vista Kernel so, hello. Can you introduce yourselves.
Rob Short: Yeah. I'm Rob Short, and I wrote the Kernel and Architecture team for Windows. The Kernel team obviously is the core piece of a system: schedules processes and finds devices, things like that.
The Architecture Team is something that I wanted to talk a little bit about, because about two years ago, we realized that we were in a little bit more trouble in terms of our ability to predict the impact of changes and to make broad, cross-group changes to Windows, and what we decided to do was have a core group of experts that would help the teams and work right across all of windows to really help figure out the impact of changes and make sure things were happening the way we'd like to see them happen, and I have some of the people with me here today. This is just a few of the people on that team. We've about six people full-time, and we have a much broader team of about thirty architects working the different groups, and they all participate as part of our architecture team but they belong to the different teams
Narrator: Okay.
Rob Short: And the idea is to really improve our engineering process and improve our quality of our engineering and be able to predict the outcome of changes that we make.
Narrator: Okay.
Rob Short: I've been in Windows for basically ever, I've been in Windows for about fifteen years. I worked on a couple of other things in between, so I left and came back again but mostly I've been working on where the hardware meets the software.
Narrator: Excellent!
Rob Short: And I'd like to introduce my next partner in crime.
Narrator: (laughs)
Rich Neves: My name's Rich Neves. I've been working here almost three years. I work on the architecture team as Rob just described, and what my responsibility or role these days is is figuring out how to police the dependency between different pieces of the systems so that we can figure out how to compose the system in a more efficient way. By efficient, I mean in a way that isolates developers from the damage they can do to other developers. So basically, Microsoft's a very innovative company, and there's people working on amazing technologies in almost every nook and cranny, particularly in Windows. The challenge we face is delivering that innovation, and what our hope is that we can make innovation itself the bottleneck, instead of delivering innovation, which has been the problem in the past, and to do that, what we're trying to do is isolate pieces of the system from each other, so that developers can know that they can work in a particular area of the system, innovating a technology, without adversely impacting larger parts of the system, that as Rob said we can't predict they're going to be impacted, and in a way that would actually jeopardize our agility in getting those features out that result from that innovation.
So specifically what we've been doing is taking every binary in the system and assigning it a layer number, which is a rank in a directed acyclic graph. There's about 5,500 binaries in the system. And what we've been doing is getting transparency now into every dependency that developers add to any of those binaries, so that we can understand what's going on. And what's falling out of that is not necessarily just the isolation I described, but also, issues. We call them, sort of, conventional wisdom ... controversies. For example, people might be thinking, well, I want to combine a whole bunch of DLL's into one DLL for perf. Well, it turns out that that's a
Re:Vista and .wmf (Score:1, Informative)
Re:Transcript up to 34 min or so (Score:5, Informative)
Narrator: Fantastic. So can you talk a little bit about what's new in the Vista kernel? So we go from XP; now we're going to Vista. So what are some of the new components?
Rob Short: A term I like to use is probably kind of politically incorrect on TV is, some of the work we do is kind of like sewers, but if we do this work incredibly well, the stuff is essential, but nobody knows that it's there.
Narrator: Yes.
Rob Short: So, if things go bad, obviously you know about it.
Narrator: Certainly.
Rob Short: Most of the work that I've been focused on for the last several years has been improving the experience where the hardware meets the software. Things like power management. We have a team of people looking at power management and working to improve how the system behaves, say a laptop for example.
If you have a laptop, how fast does it turn on, turn off, how good is the battery life? What's the experience when you dock or undock? And we've done a huge amount of work on that. We've redesigned the algorithm for hibernation so that we do a better job of figuring out which pages are already on the disk so you don't have to send more of the pages back to the disk. We've changed the way the power management interfaces to the drivers so that we have a better feel for understanding if we can just shut this thing off, right now. Today, in the older system, in XP, we actually query the driver, say, "Hey, would you, like, mind if we turn off the power?" A lot of times, people haven't coded up the driver correctly. Mostly the drivers don't care, where some really do. A disk driver, it really matters if you, you know, turn the power off in the middle of a transfer. But a lot of other things, you don't care. Mouse, it doesn't really matter that much. You know, you can go across the extreme. So we've done a bunch of work in that area.
We obviously do a lot of work in performance. One example is we had problems with heap fragmentation, and we've redesigned some of the heap algorithms so we can deal much better with much more random requests. We can deal with those and do a better job with defragmenting the heaps. So those are the types of things.
Several people--Darryl works specifically on the multimedia, and understanding how we do a better job of not having glitches in multimedia, but that also goes right through the full length of the system. It's not just buried in the kernel.
We've improved the inter-process procedure call. We have a new sort of fast, lightweight procedure call inside, in the core parts of the system. We ... stop me.
Narrator: (Laughing) He has a whole list! A cheat list!
Rob Short: There's an awful lot going on. One area where we actually make a lot of changes over time that I feel really good looking back is in the memory management area. If you think about the early NT systems, Bill Gates used to beat us up, and say, "How come you don't run in four megabytes?" And when you look at that today, and think, we're running regularly in four gigabytes today, and we have the systems in the lab that run with a terabyte of memory, the algorithms that worked back then, and the priorities back then are completely different than they are today. So we've put in work in Vista for improving the NUMA support, which is Non-Uniform Memory Access when you have a multi-processor where some of the memory is closer to some processors than to others, so we do a better job of doing the allocation, making sure that they're allocating memory that's on the CPU, near the CPU that you're running on, and then you try to run the process on the CPU where the memory actually is so you don't get cache thrashing.
Narrator: Interesting.
Rob Short: We've done some stuff for the graphics. The graphics processors today are more powerful than the CPU'
Re:MOD PARENT OFFTOPIC (Score:3, Informative)
WMF handling has been in the API of the OS since Win32 was designed. (i.e. it has always been able to inherently draw a WMF to any surface.)
However, this is not Ring 0, not even Ring 0 if you consider the Win32 Kernel as Ring 0, and in NT(XP,Win2k,2003), the Win32 Kernel is far from Ring 0 being in its own subsystem sitting above NT itself.
Just clearing up what you were saying in your post, trying not to nit pick too much...
Re:Vista and .wmf (Score:3, Informative)
Yep, although you need to be logged in as 'the' administrator for the exploit to do anything to the system.
Other accounts, even admin level ask for your permission to infect the system, so even with an open flaw, it would take the user to allow it to install. (And even some of the exploits still won't affect the system even with the user's permission with the new UAP system.)
Re:That's It?? (Score:2, Informative)
Drivers for just about any device that runs on an external bus are moving to user-land in Vista. This includes drivers for USB and firewire devices, which run in kernel-space in Linux. Video drivers are not moving to user-land in Vista, because Windows can achieve better graphics performance and GUI responsiveness by keeping them in kernel-space. You can read more about this stuff here [microsoft.com].
In any case, given that Linux doesn't even have a driver abstraction layer analogous to the Windows HAL, it's a bit ridiculous for a Linux fanboy to complain about a lack of encapsulation in the Windows driver architecture.
Re:Um...isn't vista simply rehashed NT 3.x? (Score:5, Informative)
Not to be rude, but you need to learn a bit about OSes and OS Architecture... Especially the NT Kernel and Architecture, as it somewhat unique.
NT is the underlying technology that was designed to be the low level OS. Win32 (Windows) runs in a subsystem on top of NT. The NT architecture will be around for many many years, as it was designed to be very extensible and grow to support OSes for many more years to come.
NT is the actual OS technology, Windows and the majority of the changes of Vista are in the Win32 subsystem or truly a new subsystem that is evolved from the Win32 system, as there is a new API, Graphics Model, etc.
You see, NT doesn't even have to be Windows, it also run *nix subsystems and DOS subsystems and it even use to have a OS/2 subsystem, and they all ran side by side - being equal. (Win32 got a bit of preference as it was the base Window Manager for the other subsystems. And it has more of a role for managing NT that runs underneath it.)
Even today you can download a full blown *nix subsystem and install it on any NT based OS, like Win2k, WinXP, Win 2003, Vista, etc. It will run on top of NT just like Windows does and provide you with a full *nix OS with no emulation or vitualization and yet take advantage of the NT Kernel.
As for great new OSes, 10years from now, even a full Virtual Reality based OS that has no reference to Windows itself could be released by Microsoft and still use NT technology to run the higher level new OS.
I designed the registry (Score:5, Informative)
In 1990 at Microsoft there were several requirements that drove the registry. The number of third party applications and application writers was growing very fast. Making this worse, a new object system was on the horizon which could dramatically increase the number of independently-authored "components" that needed to be registered. There was a need to store state in a segregated manner so that apps wouldn't stomp on other app's information. Also there was a "new" notion of remote manageability for the objects, so the access method should be easily remotable early in the boot process. Also the OS needed a place to store lots of very small data items.
It would have been best to use the file system, but the file system at that time was FAT which could not store small data items efficiently. The registry was the first API common between Windows 3 and OS/2 (and also NT), which was a goal at the time. Of course it quickly went out of control, since there was no rational security or ownership model. The registry was kept very very simple in order to maximize the likelihood that the next file system (either the object file system or NTFS) would be able to implement it, including in the NT kernel (which had a very simple API model). It was also the first API from Microsoft that had unused parameters for future features, such as context ids for security, query features, and other stuff. Unfortunately much of that didn't work as planned since very few applications paid attention to the requirement to set them to 0L!
I didn't expect it to be so massively overused, nor for it to survive beyond Windows 3.x. It was supposed to be superceded by an object file system (that was designed and implemented several times, but never released.)
There's a good story behind the registry, though: I designed the registry while on a bachelor party for a friend, mostly on a car ride between San Diego and Las Vegas, and faxed in the design from Las Vegas the morning after the party to the responsible program manager. Which might explain much about the design...