Linus Chews Up Kernel Maintainer For Introducing Userspace Bug 1051
An anonymous reader points out just how thick a skin it takes to be a kernel developer sometimes, linking to a chain of emails on the Linux Kernel Mailing List in which Linus lets loose on a kernel developer for introducing a change that breaks userspace apps (in this case, PulseAudio). "Shut up, Mauro. And I don't _ever_ want to hear that kind of obvious garbage and idiocy from a kernel maintainer again. Seriously. I'd wait for Rafael's patch to go through you, but I have another error report in my mailbox of all KDE media applications being broken
by v3.8-rc1, and I bet it's the same kernel bug. And you've shown
yourself to not be competent in this issue, so I'll apply it directly
and immediately myself. WE DO NOT BREAK USERSPACE! Seriously. How hard is this rule to understand? We particularly don't
break user space with TOTAL CRAP. I'm angry, because your whole email
was so _horribly_ wrong, and the patch that broke things was so
obviously crap. ... The fact that you then try to make *excuses* for breaking user space,
and blaming some external program that *used* to work, is just
shameful. It's not how we work," writes Linus, and that's just the part we can print. Maybe it's a good thing, but there's certainly no handholding when it comes to changes to the heart of Linux.
Don't piss off the boss (Score:4, Interesting)
I wonder (Score:3, Interesting)
necessary (Score:3, Interesting)
can't let shit like that fly. i think Linus did the right thing in the circumstances. i have wished no less than 1000 times over the years that my own various bosses would have shown that they even gave half as much of a crap about utterly unacceptable coding being released into production by my own co-workers.
Re:if only more technical leads had this mindset (Score:5, Interesting)
I have a "chain of clients" theory of system design.
The kernel developers protect the library developers from having to deal with the hardware.
The library developers protect the applications developers from having to deal with the kernel.
The applications developers protect the users from having to deal with all of the above.
Everyone has shit to deal with, but if you break the chain and let some of your shit spill into another layer, then someone else has to deal with way more shit than they can reasonably be expected to handle.
Re:Arsehole (Score:5, Interesting)
Finally! (Score:5, Interesting)
News for Nerds, Stuff that Matters! Thanks Slashdot! Its been a while...
Re:Arsehole (Score:5, Interesting)
>Really the tech world would be a lot more pleasant if people didn't conflate rudeness and technical competence.
The thing is that Linus is technically competent. He's in the 1% of technically competent people, so his "benevolent dictator for life" status is not changing any time soon. His ability to herd cats is manifested in his managing the Kernel for as long as he has. I'm sure if you approached Linus when he was 19 and told him that he was going to be managing one of the most important software projects on the planet for more than 20 years, he'd tell you that you were nuts.
But here we are, and Linus is 43, and he's been doing this for quite a long time and Linus *doesn't* have a reputation for being a firebrand. And honestly, to even call Linus rude ignores the fact that when he's rude, it's actually news. Unlike Theo, who is just rude all the time, so it's never news.
--
BMO
Re:Arsehole (Score:5, Interesting)
The rant wasn't about bad code, but a bad mindset. It was an apt response to Chehab's disregard for the integrity of existing Linux applications.
It also wasn't really aimed at Mauro Chehab. Even the worst manager knows better than to discipline an individual employee in front of his/her peers, right? No, that flame was aimed at the entire mailing list. Linus obviously felt it necessary to drive the point home in a conspicuous way that wouldn't be forgotten or blown off.
If there were still people at Microsoft with the technical and moral authority to send emails like that, the company's flatlined stock price graph would look very different.
Re:not good management technique (Score:5, Interesting)
No, if you bother to read the rest of the thread, the guy had a good and valid reason that makes sense
No he didn't. Sound worked, and after his patch, sound didn't work. Why would you ever commit a patch that breaks sound?
In his case, he did it because he is trying to add drivers for new hardware that behaves differently than a lot of other hardware. That's not a good and valid reason to break stuff that already works.
Re:I wonder (Score:5, Interesting)
Infrequently. I've seen somebody get hit that hard a handful of times in 20 years. I've had to do it myself twice. (There have been other lower-level chastisements.) Once I was the point man for a major reaming of a contractor that had been deceiving us for months about multiple issues. The other time was a guy that had been concealing his lack of progress on a project I was directing.
When somebody has engaged in completely unacceptable behavior, it's absolutely necessary to do at least one of two things: (1) get them off your project or (2) otherwise make them understand why such behavior cannot be tolerated. The latter is only an option if you believe the person is able and can be convinced to learn from their mistakes.
Doing it by email is a mistake and Linus can be criticized for that and, IMO, that alone. This is the kind of conversation you have face to face if you can and by phone if you can't.
Re:Arsehole (Score:5, Interesting)
It's more insidious than that.
If Linus coddles people, they get a false sense of approval. His brash, direct, and overtly emotional tone is intentional. It's to let the other party know exactly how he feels without accidentally leading them into a false sense of "oh, keep working on it and I'll approve it."
Watch that video where he flips off Nvidia.
He describes how being nice to people is the wrong approach, and makes things worse when the other side gets the wrong view, is rejected after months of effort in the wrong direction, and then Linus gets the blame for their suicide.
If he was paying them, he could fire them. He doesn't have that luxury, so he has to be clear and direct.
Don't like it? too bad. That's Linus, and he makes no excuses for how he behaves.
Re:Arsehole (Score:4, Interesting)
Hmm, the mental demands and sleep deprivation are right on target. Don't know about people trying to blow you up or kill you, but then, I haven't had a chance to work on jet control software or the like (presumably when the military's latest and greatest aircraft falls out of the sky because of a software bug, there are probably quite a few people considering killing you, especially if those onboard are harmed / killed).
This is an inkblot (Score:4, Interesting)
Re:Still.... (Score:5, Interesting)
...a better love story than Twilight.
Well, arguably what Linus is doing is a kind of tough love. He did something dumb and he's getting flogged for it. If his ego survives, he'll be a better programmer for it. That said, Linus is too much of a net celebrity to be laying someone out like that. It makes the news, like it did here, and then it makes Linux as an operating system -- and everything that runs on it and depends on it, look bad. It's like if Balmer dragged a programmer up on stage and proceeded to flog the crap out of him for ruining something in the windows kernel -- people would be all over what a horrible platform Windows is to develop for and what a horrible company Microsoft is. And Balmer could have been totally correct in everything he said about said hypothetical programmer.
There's some things you just don't do on the public stage; And an e-mail list is public, yes.
Re:Not the bug... (Score:4, Interesting)
I don't think you quite got it, as I understand it they have more error states internally than they do in the kernel API. The patch was designed to pass an internal error state that should have been replaced by the one in the kernel API before being returned but was accidentally pushed to userspace as well and the bug in the compliance tool meant they didn't notice. So far nothing majorly wrong, a poor patch and a testing tool with less than 100% coverage which caused a nasty bug to reach the first RC, a little sloppy but nothing that obviously should have been caught. The reason Linus gets so fired up is because it just doesn't matter what pulseaudio/tumbleweed is doing, there's zillions of applications out there who may be doing something crazy. If you broke one application you probably broke many more and it just doesn't matter, it's the kernel's fault.
It's been a cornerstone of Linux for as long as I can remember, you can always upgrade your kernel and all your applications will continue to work, period. New APIs are of course added but old userspace APIs are forever. It just doesn't matter how crazy the userspace code is, you don't break it anyway and even if they fixed their crazy code it doesn't solve the problem which is that there'd be two kernels with different behavior So even if he was right, he's still wrong.
Re:Arsehole (Score:5, Interesting)
Pics or it didn't happen. Apple had Asshole-in-Chief Stephen P. Jobs with the technical and moral authority to do exactly that. And guess what? OS X took almost a decade before it was anything other than crap.
That's an interesting way of phrasing your point, because of the parallels that can be drawn with Linux itself. Linux has been "crap" for well over a decade, in the sense that it has never made much headway against competing OSes in some key markets. In the past, Linux's competitors -- from Apple, Microsoft, and Sun to vendors of RTOSes that everybody uses but few people have heard of -- were all just too effective at meeting the needs of their customers.
Right now things are finally looking up for Linux. Android is proving to be a key player, if not the key player, in the most important and most promising sector in mass-market computing. Meanwhile, Apple is hell-bent on repeating every mistake from its proprietary past, making its usual heroic effort to snatch defeat from the jaws of victory. At Microsoft, there's a certified Sales Guy(tm) in charge, beneficiary of the luckiest dorm-room assignment in history. Right now Ballmer's monkey-see-monkey-do act is stampeding the company's herd of very valuable sheep, from Valve to Dell. Seismic changes are happening underfoot, some visible to the casual observer and others not.
In short, Linux's enemies are making mistakes at an utterly unprecedented rate. Leadership and vision are important, but given Apple's and Microsoft's behavior of late, it will be enough for Linus to keep his eye on the ball and his mind on the needs of the end user. The future of personal and mobile computing is up for grabs to an extent that hasn't been true since the early 1980s. All he has to do is not fuck up.
This email thread was about just that: the need to not fuck up.
Re:Still.... (Score:5, Interesting)
This little slice of linux kernel maintainer ethos indicates code quality and its twin sister, system stability, dominate the linux kernel culture as they should. Stability is the prime directive of an operating system. Its functionality must be rock solid. However this objective is enforced, be it code review, peer presure, humiliation, restricting code checkins - it has to happen. Sometimes the uber-egos of certain developers lead them to make brash decisions, and thus have to be dampened, with a club if necessary, for the good of the tribe. In the exotic world of clashing software Prima Donnas, a suit of armor is a necessity. The victim will survive.
Re:Be fair (Score:5, Interesting)
That seems a bug at pulseaudio.
Which is obviously false and a pitiful attempt to put the blame where it does not belong.
Re:Arsehole (Score:3, Interesting)
So basically what you said is you suck at managing and in fact have someone else do it for you in the form of HR.
You make the same threats statements, you just try to sugar coat them.
You just don't have the courage to say what you mean.
Re:Still.... (Score:2, Interesting)
"Manipulation" implies intent. You then stretch it to "fakery."
What's your evidence? Or, do you just have an axe to grind regarding those with more "social intelligence" than yourself?
The real problem is in the API spec (Score:5, Interesting)
Take a look at the call for which ENOENT was returned. [linuxtv.org] The API spec says "EINVAL: The struct v4l2_queryctrl id is invalid. The struct v4l2_querymenu id is invalid or index is out of range (less than minimum or greater than maximum) or this particular menu item is not supported by the driver."
This is from a generic driver mechanism for USB camera-like devices. Because such devices aren't as standard as they should be, there's an insane number of options and possible errors. The spec says to return EINVAL for both incorrect calls and calls for which the device does not support the requested item.
The problem here is that the EINVAL error status doesn't distinguish between "program made bad call and is broken" and "we're iterating through the device functions to discover which ones are available, and this ID isn't meaningful on this device so skip it" EINVAL is supposed to mean "One or more of the ioctl parameters are invalid or out of the allowed range." A correctly made call should not return EINVAL.
The alternatives are limited, though. This is related to a historical Linux design problem, which comes from a historical UNIX problem - system call errors are reported using one error code, chosen from a short list written in the 1970s. "ENOENT" isn't really appropriate. "ENOTTY" ("The ioctl is not supported by the driver"), might be appropriate, except that the usual message for that is "Not a typewriter".
The API is a rather lame and excessively complex way to return what is merely a variable-length list of fixed-format structures. One would think that Linux would by now have a generic way to do that, since it comes up in other contexts.
Here's an example of an application which seems to be crashing because the programmer did not understand the nonstandard semantics of EINVAL in this API. [stackoverflow.com]
Re:Still.... (Score:4, Interesting)
If, as a PAID developer, I submitted a patch that broke user space AND THEN blamed the user space applications and got a beat down I'd wonder wtf am I doing writing software.
That's an interesting question, since Chehab appears to have been with RedHat for well over 4 years (also bearing in mind that RedHat has a prominent profile as an "enterprise" vendor) in a senior role and seems to be pretty well qualified. While it does appear that he fucked up big-time in making over-hasty commits, breaking userspace apps, then attempting to shift the blame, I can't help feeling a bit sorry for him.
Someone with his skills should have seen the red flags and quarantined the broken code before it ever came to this performance. If he was having a bad hair day (and with no real offence intended, Linus should also be used to those), this could just be a salutary reminder that even the best of systems programmers are only human.
Re:It'll make Linux better (Score:5, Interesting)
Admins and users that are still sitting on the fence would take note, that Linus just don't take fuck as an excuse
No, instead he apparently creates a contributor-hostile culture based on blame assignment and supported by a hero model. And that is far, far more dangerous, because when your hero isn't there (and he can't be everywhere, and he won't be around forever) you have pushed away good people who could have stepped up to take over the job. Internet-famous celebrities like Torvalds are toxic to a constructive development culture that consistently makes good products over the long term.
Re:Be fair (Score:4, Interesting)
His argument is that it's a bug with pulseaudio because there are already quite a lot of existing drivers that behave in exactly the same way, and PulseAudio broke here it's already going to be broken with all of them. Subtle difference.
Re:Still.... (Score:5, Interesting)
Again, the truth hurts. Projects like these don't tolerate stupidity well, especially given the fact that it's a community effort and there has to be a lot of lack of testing and communication on the part of the responsible party when something breaks like this. The price to be paid is twofold: (1) very public expressions of the harm caused by such issues, and (2) potential relief from duties related to the issue at hand, whether temporary or permanent in nature.
I don't believe you. Someone added a sleep to the USB-HID driver in 2.6.10. The problem is they were holding a spin lock at the time. I submitted a patch for it to every single kernel release from there until 2.6.18 when a Linux kernel developer submitted the same patch and it was accepted. It was a very simple fix. If they don't tolerate stupidity, why did they ignore my one line patch and leave a huge bug in the damn keyboard/mouse (and other devices of course) for so long? I have had zero interest in contributing to these large projects ever since, as they seem to be more about ego and status than moving the project forward.
Re:Still.... (Score:5, Interesting)
He did apologize for it, quite quickly, and agreed that reverting the change was appropriate. He also asked a good question: why can't pulseaudio deal with other error codes, error codes that might be technically correct, though not in current use?
The developer wasn't being nasty, underhanded, manipulative, etc. It's possible to break user land by fixing broken behavior that userland depends on, and it's been an ongoing issue for all kernels. (Take a look at the history of the egcs variant compiler and Linux kernel compilation.)
Re:Still.... (Score:2, Interesting)
The developer wasn't being nasty, underhanded, manipulative, etc. It's possible to break user land by fixing broken behavior that userland depends on, and it's been an ongoing issue for all kernels. (Take a look at the history of the egcs variant compiler and Linux kernel compilation.)
Exactly.... Linus' suggested policy is totally b0rked.
Breakage of user programs that only worked by coincidence in the first place should not be concealed by keeping kernel bugs around.
They should test their code against the latest changes, and have a fix of their userland code out before the next stable kernel release.