Why Programmers Need To Learn Statistics 572
David Gerard writes "Zed Shaw writes an impassioned plea to programmers: Programmers Need To Learn Statistics Or I Will Kill Them All. Quoting: 'I go insane when I hear programmers talking about statistics like they know s*** when it's clearly obvious they do not. I've been studying it for years and years and still don't think I know anything. ... I have taken a bunch of math classes, studied statistics in grad school, learned the R language, and read tons of books on the subject. Despite all of this I'm not at all confident in my understanding of such a vast topic. What I can do is apply the techniques to common problems I encounter at work. My favorite problem to attack with the statistics wolverine is performance measurement and tuning. All of this leads to a curse since none of my colleagues have any clue about what they don't understand. I'll propose a measurement technique and they'll scoff at it. I try to show them how to properly graph a run chart and they're indignant. I question their metrics and they try to back it up with lame attempts at statistical reasoning. I really can't blame them since they were probably told in college that logic and reason are superior to evidence and observation.'"
Your argument is dead, Zed (Score:5, Insightful)
Maybe the problem is in your presentation. Even here, you tell programmers that you want to kill them for not understanding a topic that even you are unwilling to acknowledge mastery of. Then you tell us how hard the topic is to understand, even though you've spent so much time trying to learn it.
Is it any wonder that no one takes your suggestions seriously? You are practically sabotaging yourself with self-effacement.
These aren't homework problems you're tackling here. They are business problems and you need to sell yourself and your ideas if you want to get any traction. Do you have any evidence that your methods are better than the SOP thus far? Do you have any case studies that show how effective statistic analysis is in *any* of your projects?
Or are you simply taking something that seems like a data point and extrapolating it to cover a vast swath of applications?
Or, how about... (Score:5, Insightful)
Statisticians need to learn programming or I will kill them all.
Mathematicians just need to shutup. (Score:5, Insightful)
We know as much statistics as we need to know.
Some know more, some less. Each has traded off hours vs. knowledge in many fields.
For example: Why would a programmer who's job is to automate bean counting need to know more then basic statistics? (s)he rightfully focuses his efforts on accounting.
One post calculus statistics course gives me enough grounding to know what I don't know and punt to experts when I need to.
Fucking specialists forget all the things they don't know and only look at the world through one lens.
Re:Really? (Score:1, Insightful)
Statistics is "just" applied measure theory. Which means, among other things, that its language is Turing complete. There is infinitely much to know.
The funny thing is he's doing exactly the same (Score:5, Insightful)
He's just as arrogantly claiming that he's right and they're wrong. Now, he may very well in fact be right, but he's taking the same obstinate position the people he criticizes do.
It's important to know when your input is not desired. Even if you think it should be.
The reason people ignore you Zed.. (Score:5, Insightful)
is not because they don't understand statistics. It is because you are a dick.
Go ahead and try it (Score:4, Insightful)
It's not just statistics (Score:2, Insightful)
I've found that more than just about any other degree Computer Science and to a less extent Medical Degrees imbue the recipient with an unnatural ego when it comes to subjects with which they are unfamiliar. I propose we remove the word Science from CS degrees and call it what it is "Computer Programming and Troubleshooting". There are far too many CS graduates who think they are actually scientists.
Re:93% of Programmers Think You're Wrong (Score:5, Insightful)
A manga statistics book, eh?
I just realized I was a nerd. I looked at the table of contents and closed it down, then realized I hadn't even looked at the short skirt-wearing protagonist.
Sigh...
But to answer the article's point, elementary statistics are very easy. Advanced statistics are very hard. It's kind of like how people think "knowing the difference between circles and squares" is geometry and so analytical geometry must be just more of the same, right? It's quite possible the programmers think they know statistics because they know they're vaguely supposed to do a run multiple times, and maybe average the results or something.
It's also possible the author of the article is a know-it-all douchebag who tries to solve problems with overwrought solutions.
From TFA: "Zed: Fuck! Fuck! I have eyes! You do not! See!? No?! Exactly! Because you can't fucking see because you have no fucking eyes! Arrggh!"
Just throwing that theory out there.
sounds impossible to please? (Score:4, Insightful)
I've been studying it for years and years and still don't think I know anything.
And yet you're expecting someone whose expertise is in a different field to know more about it than you?
We can't all be experts in everything. If you're the expert in the field of discussion, get used to educating your coworkers on the topic, or find another job where you're surrounded by people with the same education and expertise as you.
The average person is an expert in no more than two or three related areas. That's why people work in teams, to cover each other's blind spots.
Re:It's not just statistics (Score:5, Insightful)
I disagree that CS is just "programming and troubleshooting", but I do agree that Computer Science is a complete misnomer. It's extremely misleading, and difficult to explain to people, "I'm a computer scientist, but no I'm not actually a scientist, instead I understand how to describe formal languages in terms of strict grammar rules and transform abstract syntax trees from one representation to another."
It shouldn't be called Computer Science, it should be called Computational Mathematics, because that's what it is.
(On the other hand, there is whole branch of CS that extends very deeply into statistics called Machine Learning, but at the core I'd say it is still more mathematics than science. There is also human-machine interaction which often goes under CS, but is actually more like psychology.. so it's not so cut and dry.)
Re:correlation != causation (Score:1, Insightful)
I think many programmers/managers would be better off with less statistics. I can not tell you the number of times I have seen a major (ie crash the damn app, corrupt data, etc) bugs go into an application because 'statistically no one will ever do that'. It is 100% predictable that someone will do it oh and then you are allowed to fix it.
Misapplied statistics are worse. Like my previous example sure out of say 10 million transactions 1 failed. But guess what? That 1 is just as important as ALL the other ones. So what if it is 1 out of 10 million that it will happen. Thats just math masturbation.
He makes some good points... (Score:5, Insightful)
...unfortunately, they are mostly lost in the irony of statements like this:
I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest.
I doubt I've seen anyone more thoroughly entrenched in a pissing contest than Zed Shaw, of the website formerly known as "Zed's So Fucking Awesome".
Re:93% of Programmers Think You're Wrong (Score:0, Insightful)
Everyone knows that 98.2% of all statistics are made up on the spot.
Re:Mathematicians just need to shutup. (Score:5, Insightful)
One post calculus statistics course gives me enough grounding to know what I don't know and punt to experts when I need to.
That's actually his argument (though I'm pretty sure he doesn't realize it, having met him a few years ago at a conference). People need to know their limits, and the strengths (and weaknesses) of others, and defer to them when they know what they're talking about, rather than talking out of their asses. As you point out, you can't know everything, but you'll defer to others who know more when you need to. I'm pretty sure Zed would like working with you based upon that fact alone (I know I value that trait and try to express it myself). Far too many people think they aren't allowed to have any weaknesses (and we all do in some area or another) so they talk a big game, and when push comes to shove, they will actively block people who actually know more than they do about the subject at hand. Working with too many people like that has driven Zed insane (IMHO) and I know I've been close to it at a couple of work places before (and really loved the one that wasn't like that hardly at all).
Re:Your argument is dead, Zed (Score:5, Insightful)
Meet Zed Shaw.
Re:Mathematicians just need to shutup. (Score:5, Insightful)
I have sat in business meetings hundreds of times where I've seen decisions made on completely meaningless and irrelevant data, because the people involved don't understand statistics. The same holds true in your personal life; decisions with purchasing products, investing money...
Now, I'll bet that most slashdot readers have the minimum amount of knowledge of statistic to avoid the most egregious errors; but more knowledge is certainly helpful. It will help you in a myriad of ways.
Re:It's not just statistics (Score:3, Insightful)
Maybe wherever you went to school they taught "computer troubleshooting" as a degree, but some of us actually got a solid foundation in the various theoretical and practical foundations of computer software engineering.
Though I do agree that "Computer Science" is a stupid name. They already have Mechanical Engineering, Chemical Engineering, Electrical Engineering, etc - why not just call it "Software Engineering"? [I'd say "Computer Engineering", but since that was my major and I also had to do transistor physics and VLSI design, it I guess does need to be separate...]
Re:Logic and Reason *ARE* superior to evidence and (Score:2, Insightful)
If you just have logic and reason, then you get religion. Logically, it worked out when it was created. There is no evidence to counter it, so it must be true. Religion was created with logical reasoning. Some may say it was incorrect reasoning, but it was reasoning nonetheless.
On the other hand, if you just have observable evidence, with no logical reasoning, you can have all the data in the world, but you will have nothing to use it with. True, you can see it, but you cannot understand why it is the way it is.
Having all of one or the other is useless.
Re:Statistics is HARD (Score:5, Insightful)
Statistics is HARD, for two reasons:
I'd argue that probability theory isn't as hard as people make it seem, but statisticians are wankers. Most of what we think of statistics was developed by people who were intimately engaged with empirical research, but modern statisticians are mathematicians, many of whom have never actually performed an experiment. They think the statistics are real, whereas experimental scientists know the truth: God made the Probability Distribution Functions. All else is the work of man.
Furthermore, modern computing has made a lot of the conceptual apparatus of conventional statistics irrelevant, as it is designed to deal with the problem of reducing problems to something that can be computed by hand and finished off with a single table lookup. Today its a rare case that we can't get at the PDFs directly, bypassing much of conventional statistics. But due to how badly the stats are taught, and how poorly probability theory is understood, we are still living in a world where p-values are the exception, not the norm, and when they are quoted they are frequently unrealistic because they are based on statistical assumptions that are not warranted given the non-idealities of the data.
So I'd argue that statistics is basically a dead field populated by zombies who are dedicated to infecting as many students as possible. If we taught thermodynamics or mechanics with equally outmoded concepts they would be really hard too.
Re:Mathematicians just need to shutup. (Score:2, Insightful)
Being socially adept is also a skill that benefits everybody but many programmers just arent. I hardly know anything about statistics, but Im not afraid to ask questions. Im sure there's stuff that other programmers know and think equally fundamental to success that Zed doesnt. It's fantastic that he's passionate about statistics. That skill certain comes in handy, but how much more important is it than helping everyone on the team get their job done, for example?
Re:Statistics is HARD (Score:2, Insightful)
The mathematics behind statistics is _not_ "Calculus 2". It is measure theory and analysis.
Summarized for people who don't want to read Zed (Score:5, Insightful)
So, since so many people don't seem to want to actually read Zed's stuff -- and I honestly don't blame you -- I'll try to summarize:
Eventually, every major science adopted an empiricist view of the world. Except Computer Science of course.
He tends to bitch a lot about computer scientists. I'm just starting a CS degree, and there is a Statistics class in the curriculum. Is he working with people with good degrees, people from a technical college with a "programming" degree, people from a diploma mill, or high school students with no degree at all?
Of course, he seems to be implying it's everyone, and doing so in a typically Zed-like way.
"All you need to do is run that test [insert power-of-ten] times and then do an average." Usually the power-of-ten is 1000...
I don't know that I've ever heard that particular statement. But it's a good point:
How do you know that 1000 is the correct number of iterations to improve the power of the experiment?
Generally because it was probably closer to a million, so I'm erring on the side of taking more, rather than fewer, measurements. But without careful consideration, I could be way off.
How are you performing the samplings?
I think this is vastly less important than how you are dealing with the data, but it is also a good point. For example, his complaint is that an average isn't enough; with detailed enough logging, he could easily go back into my data and figure out min, max, standard deviation, histograms...
How do you know that 1000 is enough to get the process into a steady state after the ramp-up period?
Not a huge deal -- the "steady state" will almost certainly be faster than the "ramp-up" period. Worst case, I'm over-optimizing.
What will you do if the 1000 tests takes 10 hours?
Either ctrl+c, or try it 10 times.
How does 1000 sequential requests help you determine the performance under load?
Very good point here. It's still a useful statistic, but you still need to measure things like 1000 simultaneous requests, not just 1000 all in sequence.
On the other hand, if your performance is acceptable with them all in sequence, you could just run it through something like Event Machine, so it's all sequential on production, too.
The most troubling problem with these single number “averages” is that there’s two common averages and that without some form of range or variance error they are useless. If you take a look at the previous graphs you can see visually why this is a problem. Two averages can be the same, but hide massive differences in behavior...
So yes, always make sure you can record enough statistics so that someone else can come along and use your data to give you something meaningful.
The moral of the story is that if you give an average without standard deviations then you’re totally missing the entire point of even trying to measure something. A major goal of measurement is to develop a succinct and accurate picture of what’s going on...
It doesn't have to be statistically accurate. It just has to be close enough.
Ah, confounding. The most difficult thing to explain to a programmer, yet the most elementary part of all scientific experimentation. It’s pretty simple: If you want to measure something, then don’t measure other shit.
This is both a very good and a very bad idea. It ties into the peeve he had before -- ramp-up time. For example:
If we want to take one single line of code and test it then we can. If we want to only verify one single query on a database then what’s stopping us?
What's stopping us is that our applications don't actually work like that.
Re:Mathematicians just need to shutup. (Score:2, Insightful)
And your 'one lens' is clueless grammarian.
Condolences to your mother for having spawned such a moron.
Re:Superior? (Score:1, Insightful)
Both, put together, are statistics.
Re:Your argument is dead, Zed (Score:3, Insightful)
Re:93% of Programmers Think You're Wrong (Score:5, Insightful)
"Lies, damn lies and statistics" is all you need to know about statistics.
This is right up there with "'click on the big blue e' is all you need to know about the internet."
Speaking as both a statistician and a computer scientist, I've seen the statistics-vs.-CS argument play out many times before, and the lack of knowledge on both sides is really striking, but not all that surprising -- both are hard subjects which take a lot of work to master. The lack of mutual respect is both infuriating and pathetic, and there's no excuse for it.
Comment removed (Score:5, Insightful)
Re:Bitch, while you were writing all that jive (Score:1, Insightful)
The reality is that a programmer who screws up the ODBC acronym probably makes less than the everyday Joe. So the challenge, offered by this 30-something everyday Joe, still stands.
Re:Very good (from someone who's taken BOTH)... ap (Score:4, Insightful)
Yup. Also, for a guy who claims to know so much about statistics and measurement, it's weird how he judges programmers so sweepingly on the sole basis of his anecdotal experiences.
Re:Zed Shaw is a tosser. (Score:2, Insightful)
Plus that hallmark observation of wise, old men in any profession: Whenever you see a power of ten, chances are the number is completely made up.
Re:Statistics is HARD (Score:3, Insightful)
1. It is not about formulas, or Calculus xxx, it is about really understanding what you are doing, and how all the formulas were derived, and some of that is really heavy Pure Mathematics in particular Algebra and Analysis, so that, if necessary, you can work out the probability theory in new situations.
2. In addition to the Math, there is Logic, Philosophy and Science in Experimental Design.
The big problem is that people who just know the formulae miss apply them to wrong experimental situations.
The most topical current example is the AGW controvesy where some Climatologists, HAD-CRU, eliminated (perceived outlier) data not realising that would mean that confidence estimators on their data were thereby faulty, so all that work must be re-done.
yea (Score:3, Insightful)
please tell me whether you would like to rely on decision theory, game theory or utilitarian techniques to handle life chances of your children or their sensitive private/critical information in a database.
Re:correlation != causation (Score:3, Insightful)
You forgot to mention that the 9,999,999 transactions are normal billing transactions, and the one that fails is the batch that actually charges their credit cards. :)
Re:The reason people ignore you Zed.. (Score:4, Insightful)
Claiming that the author is a dick is not mutually exclusive to him having a good point. The author is right in his claims that people who don't know what they're talking about often think they do and get pissy when someone claims otherwise. But the author presents this viewpoint in a really stupid manner. It is dickish to say, essentially, "Hey idiot, you're wrong", even if the person is wrong.
Note how your response is dickish, but probably right in claiming that the world is filled with arrogant/stubborn people.
Re:93% of Programmers Think You're Wrong (Score:5, Insightful)
That was kinda what I got from this. Sure, my powers of ten runs to determine performance isn't statistically sound. Did I say it was? No. Why don't I care? Because my samples are cheap. Spiking vs non-spiking is something pretty easy to see when you glance at the data.
I mean, he said we're going to die if we don't learn statistics, but he never gave a compelling argument for it.
The best example was users, but even that was lacking. If you design a script that's as aggressive on a system as a high use user and your system supports as many 'users' as students, you're safe, if it supports less you work on qualifying the problem better then.
Re:Mathematicians just need to shutup. (Score:3, Insightful)
The stats book I used in college had a table where they computed out the normal distribution equation to a table that the non calc-knowing could look up. Of course, than means that table had to be distributed on finals day.
Now, there's a funny think when you write out a table of values. You have to make an intentional mistake, or you're not able to have an effective copyright because the infringer could claim they did the work themselves.
I wrote a computer program to check the values to four digits (because that was the precision of the table) and found the one mistake. Funny thing, there were people who believed everything in the book had to be perfect... they also seemed to each have a favorite religion book, but the people didn't agree on the same one. The professors were alarmed... they had a problem about to use that value planned for the final...
Re:93% of Programmers Think You're Wrong (Score:2, Insightful)
Re:93% of Programmers Think You're Wrong (Score:2, Insightful)
He has got a point that Computer Science graduates do value logic and reason (or less charitably bullshit) over evidence and observation.
In fact one of the best CS books I've ever read was "Computer Architecture. A Quantitative Approach" by Hennessy and Patterson precisely because all the rules of thumb in it were backed up by measurements.
Then again most CS types have realised that with a bit of Google assisted cherry picking of the statistics they can pretty much prove any of their preconceptions to be true, i.e. their favourite ultra high level language just happens to be "potentially just as fast or faster than C++, the problem is that most people don't have the skills to do it". Sigh. It's one thing to say you like a language subjectively and are more productive in it, quite another to claim it is fast when most measurements say it is just isn't.
Re:The reason people ignore you Zed.. (Score:2, Insightful)
Ah, the good old clash between the real world and the way you(we?) think it should be.
Pointing out that people are wrong is a sensitive process. If you do it the wrong way, you provoke an emotional response that stops the person you're trying to convince from absorbing what you're trying to tell them.
It doesn't matter if you're right or wrong, if you convey your information in a way that is perceived as "being a dick" it will never reach its destination. That sucks, but it's just the way most human beings work. And I very much doubt that this is a root cause of "what is wrong with the world today", unless people getting pissed off because some know-it-all jackass is telling them they're a moron is a recent development.
Re:93% of Programmers Think You're Wrong (Score:5, Insightful)
Thing is: You can only be expert in ONE of them. Period.
Hundreds of cryptologists prove you wrong.
Re:93% of Programmers Think You're Wrong (Score:3, Insightful)
Thing is: You can only be expert in ONE of them. Period.
You can easily be expert or well informed in more than one field.
I for one, choose CS. Waaayy more interesting, and compared to the nerdiness level of statistics, we look like Joe Sixpack coming to the club in his sports car, with two girls in the back. ;)
If I want to do statistics, I can always hire someone.
I suppose if I really want programming done I can hire someone. There problem you have here is trusting the person you hired to have done their job properly so you want to have some understanding of what is actually required.:)
If you are a consultant you have to have an understanding of all the fundamentals that are required to get the job done. You don't have to be an expert in all fields but you have to be able to communicate with the people that are giving input and if that requires learning what can sometimes be a difficult field then so be it.
Any type of computing requires knowledge of "Numerical Analysis", "Statistics and Probability", "Logical thought" and surprisingly "Art". You also should be open to input from a wide variety of sometimes conflicting ideas and have to the ability to determine what is the correct solution rather than just a solution as well as having the ability to reason and sometimes compromise with all parties. This is actually called human communication (sometimes diplomacy) and no one would say this is an easy thing to do.
Re:93% of Programmers Think You're Wrong (Score:4, Insightful)
You can only be expert in ONE of them. Period.
[shrug] Depends on how you define "expert," I suppose. I have one MS in CS and another in biostatistics, and am currently working on a PhD in bioinformatics, where I use the knowledge I've gained in both fields pretty much every day. If you think CS is "waaayy more interesting," that's fine for you; personally I find them equally interesting and valuable.
Re:The reason people ignore you Zed.. (Score:2, Insightful)
Zed sure embarrasses himself by writing such an atrocious piece of garbage.
Maybe people would listen to Zed if he didn't:
a.) Depend on vulgar language to emphasize an argument (and subsequently)
b.) Prove himself as a huge douchbag.
Re:93% of Programmers Think You're Wrong (Score:3, Insightful)
Re:Your argument is dead, Zed (Score:4, Insightful)
That's one of the reasons working in IT is not all that satisfying [computerworld.com]. Many problems have multiple solutions which for the most part are equivalent in function but vary on what they're attempting to optimize for (* see below) yet developers seem to latch onto the solution they thought of and become down right rude and nasty when evaluating a teammate's solution. When every developer assumes he is the smartest of the bunch and all others are morons it fosters an environment where everyone is unwilling to compromise and a 3rd person usually has to step in to break the tie. That leads to a hostile work place where thought battles frequently occur. Losing a battle causes a teammate to become afraid of undue criticism in the future, so the next time around they over engineer the code trying to cover all bases. This leads to large systems that solve fairly simple problems with overly complex implementations. After a few cycles of this, the software is unmanageable, which becomes evidence proving to the developer that his teammates and ones who came before are idiots with no clue, and now it is up to that lone hot shot to bitch about fixing the mess, which of course is accompanied with many nasty critiques and insinuations.
I am a developer with a fairly open mind and I strive to eliminate ego from the workplace by staying on the positive, helpful side, but honestly I'm getting sick of working with people who don't try to do the same.
* Example, solutions can be optimized to target maintainability, readability, CPU/IO performance, availability, reliability, correctness/precision, recovery, automation, reduction of complexity, extensibility, cross platform, resilience to change, parallelism, security, partitioning, modularization, popular design idioms. The list is nearly endless.
is this really necessary? (Score:3, Insightful)
I'm a physicist, I know plenty of statistics. The kinds of statistics he's talking about are not hard. If you can do algebra, you can do things like calculate the standard deviation and variance of a set of measurements.
Was this rant really necessary? I run into people in physics who don't take care of these details. I find that a simple "can you put a standard deviation on that number?" or "can you repeat the experiment?" generally gets the job done. If you want to be more scientific, just start with those questions, and see where it takes you... you could even add "please" if you wanted to be nice. I find threatening people with death and belittling their intellect while talking about trivial calculations doesn't generate useful data.
To be fair, it sounds like Zed has been working as staff at a university. This has nothing to do with statistics, but it's probably the real reason he's in such a bad mood.
Re:93% of Programmers Think You're Wrong (Score:3, Insightful)
Standard deviation is useless if you're not working with samples that have a normal distribution.
Why? No. You can measure the standard deviation of any distribution, normal or not. And it is what it is, independent of distribution, it tells you how much you should expect samples to deviate from the average.