Hiring Programmers and The High Cost of Low Quality

An anonymous reader writes "Why is it so hard to find good programmers? And why should companies favor hiring fewer more senior developers rather than many junior ones? Frank Wiles discusses his thoughts in his article A Guide to Hiring Programmers: The High Cost of Low Quality"

Internal Inconsistency in his Argument

[mosel-saar-ruwer]

Anyone who has been a developer or managed developers can tell you that an expert can accomplish as much as 10 average developers. However, companies typically pay only a 10-20% premium for an expert over the average programmer. Whether or not their title is Lead, Architect, Development Manager, Guru or whatever nomenclature the company uses. I am not saying that if your average developer is paid $50k/year that you should pony up $500k/year for an expert. The employer/employee relationship never works like that, but what employers don't seem to realize is that in the end paying more saves them more.

This guy should be ready to put his money where his mouth is: If there really is such a thing as the über-programmer, who is literally 10 times more efficient than his average [median] colleague, THEN BY ALL MEANS he ought to be paid 10 times as much in salary - maybe even more.

The very fact that such disparities in salary don't exist means that either the über-programmer does not exist, or else there is something so screwy about the internal politics of corporations that the suits in management won't stand for some dweeb hax0r making ten times their salaries.

But I think that if the über-programmer really does exist, then eventually the free market will figure that out, and compensate him accordingly.

Re:My own experience...

[Anonymous Coward]

Instead of only hiring cs people with degrees in engineering and cs, broaden your horizons. You may be pleasantly surprised at what someone who has learned programming from a different background can bring to the table.

They exist, but they don't know it.

[Anonymous Coward]

Here's what Paul Graham had to say about Great Hackers:

Because you can't tell a great hacker except by working with him, hackers themselves can't tell how good they are. This is true to a degree in most fields. I've found that people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent.

http://www.paulgraham.com/gh.html [paulgraham.com]

Re:Sigh.

[nurb432]

But the advantage of being a talented generalist is you have a N+1 higher chance of remaining employed then someone that can only do one thing, no matter how well.

Re:My own experience...

[Schnoogs]

The thing is, is that 99% of the people who put in their resumes are CS graduates. I guess we're only as good as the pool of talent that expresses interest in our openings. I understand what you're saying though and you might be 100% correct.

Re:Internal Inconsistency in his Argument

[networkBoy]

I worked for a company that got bought by a bigger company.
We had an über programmer. He left because rather than exceptional pay, he wanted good enough pay and a small company style work life.

We then got in a pickle where some kernel mode drivers for NT4 needed to be revised and SoftICD'd in. Even though he doc'd everything and gave training to our programming staff about gotchas and pitfalls as well as maintenance, it was something that only about 100 people in the world could really do. All we could get done is a widely variating series of BSODs. We hired him back at $12K/day + travel for 5 days of work. He did work his ass off, further document everything, and provide additional spot training to our two brightest. The job was done and he had a check for $60K. I suspect that the training and docs took the vast majority of his time. I asked him why so much (and why MegaCorp would pay that) and he said it was simple. They were a big company not interested in paying him either in lifestyle changes or money so he didn't stay, but for a short job he charged what it was worth.

The free market did work. (considering his solution was cheaper than a contract with MS for the same work by $40K).
-nB

Re:They exist, but they don't know it.

[nomadic]

Here's what Paul Graham had to say about Great Hackers:

Because you can't tell a great hacker except by working with him, hackers themselves can't tell how good they are. This is true to a degree in most fields. I've found that people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent.

I've got to disagree; in my experience hackers tend to be pretty damn egotistical. Average hackers think they're good. Good hackers think they're great. Great hackers think they're on a higher plane of existence.

Re:hireing more people is better then over working

[chicagotypewriter]

Your comparison seems to be of people of the same level. Two new grads vs. one new grad, or two seniors vs. one senior. To change your original scenario, would you rather hire two recent grads at $50k, or one senior level worker at $125k? It is $25k cheaper to hire two recent grads...but when you put them together, do you get the quality of a senior level worker with say 8 years of experience? Probably not. The one senior level guy might work 50 hours a week, but he probably does more than two recent grads 3 months out of college working 40 hours.

Then again, I didn't RTFA, just other comments.

Ask the Yankees if this works.

[Anonymous Coward]

The New York Yankees have the highest payroll in baseball by far. $195,229,045 according to ESPN. $52,105,331 more than second place (Boston) and $112,176,945 higher than average. Yet their current record is 62-50, 6.5 games back from Boston. Even the lowly Arizona Diamondbacks have a better record of 63-50 with a payroll of only $52,067,546.

Baseball and programming might not be exactly alike. But look at it this way: The most "talented" individuals in baseball get the most money - yet all that talent pooled doesn't result in the super team that this guy is dreaming about. In baseball, it's fairly easy to measure past performance. Stats up the wazoo, in addition to recordings of their job performance that are easily available. But programmers aren't that way at all. Even if you try your best, filter like crazy, do interview after interview after interview, test after test, you may be getting a complete dud - shows up to work day #1 drunk and starts to watch graphic pr0n.

HR could work from now until the end of time trying to find the perfect fit. They could spend a billion dollars per employee trying to find the best. But that's still no guarantee that you'll get what you're looking for. Usually, you can't know the quality of the service until at least the start of when it's being rendered - and often times not until many years later.

Re:Internal Inconsistency in his Argument

[DavidHumus]

He's absolutely correct.

There are numerous studies to support this. Taking about 20 seconds to look for one: http://www.joelonsoftware.com/articles/HighNotes.h tml [joelonsoftware.com].

Also take a look at "Code Complete" by Steve McConnell and "Peopleware" by DeMarco and Lister. Actually, I've seen credible estimates of a factor of 25 times productivity between best and worst programmers. Given the negative productivity I've witnessed, even this may be an under-estimate.

This should be a well-known fact but it isn't. A major part of the problem is cluelessness by the people doing the hiring. You probably can't scale salary by productivity but how about something like the square root of productivity? Of course, the hiring of Bob Nardelli, the mediocre CEO who did nothing at Home Depot, by Chrysler shows how unrelated salary and effectiveness can be.

Easy answer.

[Estanislao Martínez]

As a generalist programmer, originally trained in cognitive science, who has formerly worked in several disparate industries, was a systems administrator, programs in half a dozen languages (including perl), etc, etc...Apparently I'm supposed to be making twice my salary. Goddamnit!

See The Market for Lemons [wikipedia.org]. The existence of tons of bad programmers, and the inability of employers to tell them apart from good ones, drives the salaries of all programmers towards that of the average programmer.

Read Joel's book

[AxelTorvalds]

Smart and Gets Things Done.

I'm a little dismissive of the mystique around the required "super hackers" that never need to look for work but there is a ton of great advice on just hiring people.

I'm an engineer. Been there and done a lot of that. I'm not going to say I'm one of the greatest but not too shabby, I've built some stuff and made some good money, you, know left a few marks.. As a more senior guy I've been taking on more and more leadership and to be completely honest, I like to think there are things I know to look for and catch, but I suck at hiring and team building. At the end of the day it's about building products, selling them and making money and the balance between people you think are a good personality match vs. the people that are technically good enough vs. people that are actually motivated and want to work and be successful is hard. We've hired folks we though were good personality matches for the team and turned out to be terrible technically and completely unmotivated, as much as you might want to like them, you simply won't when they are trying to play big business "CYA" games and not actually contributing to the team.

I'm kind of dealing with a situation now, we're a small team, 4 or 5 developers and 2 testers. We hired the 5th developer based largely upon a recommendation from one of the testers. He's a marginal tester to be honest, good guy, just not super motivated, why we hired his recommendation is looking more and more stupid by the day, we value recommendations. After reading Joel's book I've found like 6 or 7 indicators that probably would have flagged this guy that we simply didn't think about. We were in a hurry, we thought the req would go away, etc.. Honestly, I'd rather have one fewer people and better morale that this guy, seriously in 6-8 months of having him, I cannot point to a single substantial contribution. Now we get to go through the process of firing him which sucks for every one involved also.

Basically, you always want smart people, you want motivated people, people that do a good job, people that have some passion, good communicators, strong team people that know what it is to be on a team, you want all of that stuff, all of the time and it's hard to find. We pretend that parts don't matter or don't matter as much. Having shitty people on your team just flat out sucks, doesn't matter how good everything else is.

Re:Internal Inconsistency in his Argument

[AuMatar]

Not really. The uber programmers rarely have the skill set needed to found a buisness. Those tend to require high people skills and financial skills, which are almost completely disjoint from the programming ones. Plus it would mean spending time doing all that bullshit, instead of programming.

The real answer is that most uber programmers work for that small premium. They get to do what they want, get a few other perks like their choice of assignments, and are generally happy as is. Money isn't really the key motivator for them- if it was, they'd be in another field that pays more.

Unfortunate reality

[ravyne]

On the flip side, why is there such an excess of not-so-great programmers out there? The answer is simple: The higher education system is turning out not-so-great graduates. In an ideal world they would not, but we live in a world where there are "CS Graduates" who have never seen anything more than pseudo-code and java. There are some great programs and great graduates to be found for sure, but I think the writing on the wall is apparent -- the average graduate is a below-average programmer. There needs to be more hands-on exposure to real, complex code, or better yet, production code.

In the interim, unfortunately, we realistically need to take in some of the graduates that we have and finish the education they apparantly never received in full. If we don't -- if we let a bubble form between now and when the educational system corrects itself, then we will effectively lose much of the "tribal knowledge" that is passed down through the generations of the workforce.

You cannot sustain a class of experts in any endeavor simply by surrounding them with other experts. At some point they must mentor or pass down their knowledge to the next generation -- but the best way to ensure the next generation is to make sure that they're at least on-par as a developer.

I say all this as a relatively young developer who graduated in Computer Science in 2002.

This article has no value

[Anonymous Coward]

I disagree with the premise of this article. This article assumes that it's easy to determine who is and who isn't a good programmer and then goes on to discuss the benefits of choosing a good programmer over an average programmer... I find this discussion meritless because its first assumption is, in my view, plain wrong. It's difficult to hire good to great programmers because generally, it is virtually impossible to determine how good a programmer really is based on an interview, a resume, the person's current salary at his/her existing job, or references. It is this fact that it's so hard to tell who is good and who is not that makes hiring difficult. Hence, you get the statistical mix of good and bad programmers after you've hired enough people. If it was easy to determine the quality of each programmer, don't you think everyone would prefer to hire the better programmer? Hence, I think the rest of this article is rather moot. Maybe a more interesting article would be an exploration of how we can reduce the costs of trying to figure out who is good and who is average to assist those who are in the market to hire programmers.

Testing....

[lysine]

I work for a small game developer. We recently announced we are developing a pretty big title, and some fanboys of the game said that we'd make a lousy product because our Programmer Job description on our job page reads like this :

If programming games is your dream, we're the place for you. Please send in examples of your work, no matter how trivial. Websites are acceptable as well. You can't always work in one language long enough to know the syntax by heart. But the concepts are reusable, that is what we look for. C/C++ knowledge is an advantage but not necessary.

The comment went along the lines of, how can a company make any good games if they hire just anyone off the street? Well, we do. But everyone has to pass a test. We basically hand them a copy of the engine and give them the instruction of. "Complete it" It's their job to read the code, figure out what it does, and add their own code to extend functionality. The test basically tests their ability to grok it. To get it to compile. Then to extend the code while following proper standards and naming conventions( As long as you follow the style of the rest of the code, we're happy ). Finally, their creativity. We don't tell them how to complete it. So some people do the bare minimum which shows competence, and some go all out. But usually you can tell what they like. Some are AI heavy, some do a lot with player mechanics, and some start extending the engine when they want to do more (Warning : Not Invented Here Syndrome Probably Present).

So we've got some older guys from the VIC-20 days, some young college grads and some non-traditionally trained programmers. I'm a tools and build process guy myself. Engine is another set of people that hate muddling with gameplay because you can only spec so much, the rest is up to...testing and feel, and they hate repetitive work like that. The gameplay guys that love to noodle and pull magic numbers out of the air and test until it feels "just right".

We also do products in staggerred teams. Several experienced developers, several inexperienced. Let the experienced funnel the knowledge down. Rotating R&D cycles, so everyones fingers is in the engine at least a little bit. Really experienced outside developers have their issues sometimes as well. They are very set in their ways, and it's hard to mold them to your system. Which is why Microsoft prefers hiring physics and math majors. They haven't learned any bad habits yet. Sure when it comes to crunch time, we do neglect the juniors, but that is when the smart one's start to shine.

Re:Languages

[gatesvp]

I actually got a recruiter e-mail the other day for which I was basically a perfect match. I even had the soft skills they were looking for "bilingualism", training and support experience.

So I replied with the resume and comments about salary range (which wasn't listed), experience range (also not listed) and how the job listing looked very "boilerplate" and looked like a request for "bodies" and not for talent.

He replied with the list of 20 questions asking for further details. The first 8 questions were "how much experience have you had with technology XYZ"... So I replied by saying: you didn't answer my question about salary range and you asked really stupid questions, clearly this is not a job for me.

End of the day, recruiter just cost them an opportunity and never addressed any of my issues.

Re:Sigh.

[Heembo]

The problem is that you are a admin/script writer. You admit this clearly by touting your know-how of PERL of all things.

When folks ask, what do I write in? I say, Java (since 1.0.2, I'm now into 6), HTML (really, XHTML 1.0) CSS/JavaScript (my AJAX runes on even IE 5), etc. I have Software Development experience ranging from big teams at GE to small elite teams. Right now, I can make around 150k easily/year. I know Perl, PHP, etc. But I admit to these last.

The moment I'm not useful I know they will fire me that day. But right now, these specific skillz pay very well.

Re:How do you tell the difference???

[nelsonal]

My field is finance where you have the same problems (it's very difficult to tell the excellent from the good (and even sometimes the average). One tool used is a very bonus loaded compensation structure. You pay everyone decently well (higher than average but low for the good and excellent) but with the common knowledge that the very good will receive a bonus that is many times their salary (after the race). It's expensive (big banks pay out half of their revenue in employee costs) but they seem to continue to attract and retain (although it bounces from firm to firm pretty regularly).

The trouble might be that it's more difficult in coding to tell who really won the race, even after the fact.

Re:Sigh.

[ktappe]

the employees forming close, almost family-like ties, but management was completely insane. That's where I discovered that being the guy that is considered absolutely invaluable doesn't actually insure job security

Wow, it's good to hear I wasn't the only one. In the 40-person company where I worked as the sole I.T. guy, the owners were also insane. I got booted after seven years of service because the owner's wife took a dislike to me (I couldn't fix the inherent problems with the horrid software she chose to buy without consulting anyone, she didn't understand why I had the desktop publishers using Macs, etc.) That's (micro)politics too, so it's dangerous to assume small companies don't have that problem. Now I'm at a Fortune 100 company where you essentially have to get arrested to get fired, though such job security is compensated for by the endless red tape required to get anything done. Everything is a tradeoff.

Re:Sigh.

[jlarocco]

I think we're talking about two different things.

By "specialist" I wasn't talking about idiots who specialize in one part of a particular programming language. I was talking about the people who specialize in writing software for one part of a particular industry and know that part of their industry very, very well.

For example, people who specialize in industrial robotic systems, telecom systems, automotive computer systems, avionics systems, etc.

To put it another way: Do you want the avionics in the plane your flying in to be written by 10 specialists with 200 years combined experience writing avionics systems, or by 50 general programmers with 5 years combined experience writing avionics systems?

Re:Oh god, they never get this right.

[allanj]

This is *exactly* what I think is best. Let the experienced hands create the design and let them solve the difficult stuff. Then have the juniors "fill in the blank" and help them along the way. Occasionally have them participate in design tasks to give them some experience and a little more of the "why" part about the design - this will reduce the number of annoying questions tremendously afterwards.

As you can probably tell, this is what I do - design and difficult stuff. I really enjoy that. Others do GUI, build process, configuration, and what-have-you. Some of them like their work because they get to go home and not think about their work until next morning. They will stay "junior" forever, but are perfectly happy with that. They don't have a passion for their work, so they should never ever be allowed to do the design or solve the difficult problems, and most of the time they don't even want to.

Now management, that's another issue. They simply fail to understand that there can easily be a factor of 5 or 10 difference in the total time it takes for any two of us to complete a certain task with any reasonable measure of quality. This works both ways - I end up spending more time doing GUI work than those who like that work, simply because it is a kind of work that does not motivate me. Likewise, solving that OS and transport-independent highspeed failover service protocol was my work, because that is what I do 10 times faster and better.

Re:More to it than skillz

[Echolima]

Girls like guys with skillz.....nunchuck skills....computer hacking skills...you know, guys with skills...

Re:Sigh.

[Anonymous Coward]

yea like when your 120k per year Oracle Certified Administrator(tm) can't install Oracle on a Sun box and doesn't know how to install java either, which is required by the Oracle Installer, which is written in java. How the fuck can you be an "Oracle Certified Administrator" and not be able to install and configure the product on the primary platform it runs on out in the world?

Frightening isn't it? Certification 4tw! I stepped in and had his installer running in 7 minutes after he wasted 2 days on the phone with support. Keep in mind this was Oracle 10i and I'd never installed it on Sun before. I'd never been at a Sun prompt, but know BSD, Linux and Unix systems. ... Stupid ****'s I can't believe that guy has a job, period. Any time a developer has to step in and do an administrators job for them, be afraid, very afraid. A couple of years later the PM was wondering why the project he spent 1.2m on failed...

It's almost as bad as the MCSE I worked with who didn't know what ftp was. She literally couldn't get to Microsoft's ftp server to download something (this was back when their patches, resource kit, etc. were available on ftp)

If I'm ever in charge of hiring, there will be a system in my office when I interview people, where they get to show me what they know instead of telling me about what they know, or showing me a piece of paper. If it's an Oracle Admin, there will be a default install of Sun OS on the box, and the candidate will install it right there in the interview and get it up and running in front of me, or he'll look elsewhere for a job. Just take a sliver of their supposed skill set and spot check.

Only takes 10 minutes right? It's not hard to see who the idiots are, just put a terminal in front of them and ask them to do something.

-AC

Re:Sigh.

[djasbestos]

I work for a small company (~50 in the corporate office), and top management refuses to flat out fire our contractor despite mountains of incompetence because we've got too much money into them and their shit software that does not work very well, if at all.

I'm somewhat of a generalist (minus pretty web GUI's...), and my boss and I agree we can outdo (performance- and time-wise) the contractor through the entire cycle.

FTA: "Experts use better tools and care deeply about their craft. They aren't assembling bits on an assembly line, they are crafting a unique product to solve a unique problem. Experts are lazy, they work smarter rather than harder. Experts prefer the easiest solution that gets the job done. Experts aren't interested in creating complex solutions simply to have the complexity, that misguided egoism is the territory of more junior developers. They often get it right the first try and almost always on the second one."

This is the sad truth of my job: replace "experts" with my name and "more junior developers" with "our off-shore developers led by an American" and you have a pretty solid description of our development story. Locally, I *am* the dev team, and that doesn't really bother me as we aren't an IT company. My boss and I agree with this article: we should axe the clowns overseas and get one more *good* developer in here.

Then again, having these bozos arounds just makes me look even better, but it's not worth the headaches, and I'd frankly rather just write ALL the code and make it bulletproof and completely end-user configurable and operable. So in the words of the GGGGP: sigh.