Hiring Programmers and The High Cost of Low Quality

An anonymous reader writes "Why is it so hard to find good programmers? And why should companies favor hiring fewer more senior developers rather than many junior ones? Frank Wiles discusses his thoughts in his article A Guide to Hiring Programmers: The High Cost of Low Quality"

Sigh.

FTA: ...Experience is key, but not necessarily in ways you might imagine. Time in the saddle, with a particular language is not as important as diversity of experience. Someone who has worked in several disparate industries, a generalist, is often a much better developer than one who has spent years in the same industry. There are exceptions to this, but in general I have found this to be the case. Bonus points if your developer was a systems administrator in a former life.

Some of the best developers I know were originally trained as journalists, mathmaticians, linguists, and other professions not normally associated with software development...

As a generalist programmer, originally trained in cognitive science, who has formerly worked in several disparate industries, was a systems administrator, programs in half a dozen languages (including perl), etc, etc...Apparently I'm supposed to be making twice my salary. Goddamnit!

*stomps off in search of his boss*

These days, being a programmer generalist (even worse, one with admin experience) just increases the types of shit that get dumped on you...Where they might have had to hire a person to do the front end GUI code, a person to do the database work, a person to set up the server, and a person to code all the services that need to constantly run in the back end, instead, since they've got you, you can do it all, while the specialists sit around drinking coffee and making catty comments about how much better they are at what they do than you are.

My advice is specialize in something to the point where when you do any work on it, it's immediately out of the comprehension of a generalist or a less accomplished programmer...Sure, everyone will hate you, but they'll have to deal with you, and you'll be in a position to dictate terms. What's a generalist got? They're great employees. Big deal. Being a great employee is like being a great dog; at the end of the day, they'll still euthanize your ass when you're no longer of use.

//Not bitter or anything.

Re:Sigh.

But the advantage of being a talented generalist is you have a N+1 higher chance of remaining employed then someone that can only do one thing, no matter how well.

Re:Sigh.

Tell me again? Just how is it you've managed to get this far in life having never fallen victim to office politics?

Three possible methods... may be used in combination:

1) Small companies/organizations
2) Being completely oblivious to politics and not getting involved
3) Consulting/contract work

Note, I'm not the original person you were asking. I just thought I'd chime in.

Re:Sigh.

Well, the question wasn't how can anyone do it, the question was how did *I* do it. The first point, avoiding large corporations, is probably the most effective way that I have avoided office politics deciding the fate of my job. You can form more personal bonds in a small company and it is much more difficult for someone else to hide their incompetence.

Another good strategy is simply to be good at what you do and don't give anyone any reason to doubt your sincerity or integrity. Always be upfront, frank, and honest. Never be afraid to say "I don't know" if you don't know something. If/when someone approaches your boss to complain about you (presumably for no good reason), your boss will take your side by default and you can therefore remain oblivious to the politics.

Of course, if you're in management, too bad. Politics is pretty much your jobs then. ;-)

Re:Sigh.

The first point, avoiding large corporations, is probably the most effective way that I have avoided office politics deciding the fate of my job. You can form more personal bonds in a small company and it is much more difficult for someone else to hide their incompetence.

My personal experience (clearly being a statistically significant single datapoint . . . ) was in two small business (one around 20 employees, the other around 40 employees), and both were extremely cut throat. In particular, the second one had the employees forming close, almost family-like ties, but management was completely insane. That's where I discovered that being the guy that is considered absolutely invaluable doesn't actually insure job security, but rather makes you a target by people who consider you a threat to their own position.

Now I work under a consulting company under a Fortune 500, where I'm almost completely insulated from the normal office politics. Whenever I have a bad day, I watch "Office Space" and remember why I'm so lucky.

Re:Sigh.

Avoiding being a victim to office politics is doable. It is about making yourself look good as well as the rest of the department. Politics come in when you are trying to make yourself look good either by focusing completely on yourself or at the expense of others. If there are other people trying trying to make themselves look you you help make them look good, if they are trying to make you look bad you make sure you still look good without making the other guy look bad. I work in a small company but I am one of those evil contractors who do work for bigger companies and there are always people want to see me fail but normally after I help them succeed then they are normally more welcoming to me.

Re:Sigh.

"But the advantage of being a talented generalist is you have a N+1 higher chance of remaining employed then someone that can only do one thing, no matter how well."

Not so true: say you are quite expert on A, B and C. On a mature or stressed market that only will mean that you won't get work neither on A, B nor C because those job positions will go for "real niche expert on A", "real niche expert on B" and "real niche expert on C", repectively.

Re:Sigh.

Says you. I'm a "generalist", but damned if I don't solve tons of problems around the office that stymie specialists because they don't know the full system, or how to think of it from any other angle than what they've been trained. As in, the Java interface programmer who keeps asking me how to use LDAP, which he hasn't had experience with, or why the 64bit version isn't working. Generalism still has a lot of place in a business.

Re:Sigh.

I think we're talking about two different things.

By "specialist" I wasn't talking about idiots who specialize in one part of a particular programming language. I was talking about the people who specialize in writing software for one part of a particular industry and know that part of their industry very, very well.

For example, people who specialize in industrial robotic systems, telecom systems, automotive computer systems, avionics systems, etc.

To put it another way: Do you want the avionics in the plane your flying in to be written by 10 specialists with 200 years combined experience writing avionics systems, or by 50 general programmers with 5 years combined experience writing avionics systems?

Re:Sigh.

I would say an expert can at times generate up to 3 times a much output, but 10 times is ludicrous.

Heh, quantity is not quality. Do you have any metrics? DeMarco and Lister [amazon.com] do, and their data seems to show 10x. As in, not 'more code' but 'better code, fewer bugs, faster execution'

Re:Sigh.

My advice is specialize in something to the point where when you do any work on it, it's immediately out of the comprehension of a generalist or a less accomplished programmer

Perl and Batch files it is!

Re:Sigh.

Not specific enough. Just batch files!

Re:Sigh.

Every five years someone rediscovers, The Mythical Man Month [amazon.com] and thinks they've had a great insight. People should be handed a copy of this when they start their tech jobs. Managers should have it inserted forcefully into appropriate orifices. Hardback copies for senior management.

Basically, some people are just better coders, and adding sub-standard assistance just ensures late, sub-standard software. Adding people to late projects makes them later.

Re:Sigh.

^^ No kidding, it's like this guy just read about the Code team as surgical group. I'm a baby programmer (young, i don't actually program babies) and even i think this is old news.

How do you tell the difference???

While many people have an intuitive feeling as to what constitues a Good Programmer from a Bad Programmer, there are very few quantitative measures. Bad software does not look vastly different from Good Software.

By some estimates, Good Programmers can be a factor of ten or more productive than Bad Programmers, yet they are seldom paid more than a few tens of % higher. It would be far better for most companies to pay double the going salary to attract only the best, but unfortunately business thinking does not seem to be structured that way.

Most organisations base their planning on some convenient notions like programmer-months etc, using some standardised measure for programmer capability. These measures are great because they make the spreadsheets look neat and tidy. They also make all the outsourcing logic work: "I can get programmers in country xxx for $10 per hour". Untimately they are flawed because you get what you measure. If you don't pay a premium for good programmers you won't get them. You end up spending mucch more on crappy programmers.

Re:How do you tell the difference???

Yeah, being able to tell which programmers are the good/great/uber is HARD. It's much easier for companies to go on metrics as above instead of attempting to filter through for the excellent people, or even the most relevant person for the position.
Compounding that, it's rare that a coder will admit to being subpar. Chances are even if you're dailywtf material they think they are great programmers! I've been doing code in one form or another for over a fifteen years and consider myself pretty good, great sometimes. I've worked with one uber programmer in my entire career (John Kichury of SGI), maybe 2 or 3 others that came close, but have met many that act and talk like they are. Following on their projects always has a common thread of being overly clever, loosely documented and hard to maintain.

Re:How do you tell the difference???

"It would be far better for most companies to pay double the going salary to attract only the best"

1) Everybody knows that some horses run faster than anothers. The problem, my friend, is telling appart *which* one will run fastest this evening's race.

2) Do you really think that by paying double bad programmers will be repeled and won't try to apply for your job offer?

find the enthusiasts

I think the single best way to find good programmers is to find enthusiasts.

The reason is that it's easier to determine how enthusiastic someone is than how good a product they develop:
    - enthusiasts usually have side projects
    - enthusiasts often create libraries of code that can be reused
    - enthusiasts will have a variety of favorite tools - and can explain why they like them
    - enthusiasts will likewise have a variety of favorite methods - and can explain why they like them
    - enthusiasts read widely in their field
    - enthusiasts know the names of those who have made impacts on their field
    - enthusiasts often find themselves putting in too many hours - because they *enjoy* the work
    - enthusiasts gravitate together - put them in a room together and you'll have a lively conversation

And there are technologies, methods and tools that attract enthusiasts. For example, I've found that even if python and ruby aren't the most marketable languages out there - they are great ways to find the enthusiasts.

Of course, this won't help a manager that lacks enthusiasts on his team. But a technical manager who is himself an enthusiast, and builds such a team should be able to easy find more. At least in my humble opinion. :-)

Easy answer.

As a generalist programmer, originally trained in cognitive science, who has formerly worked in several disparate industries, was a systems administrator, programs in half a dozen languages (including perl), etc, etc...Apparently I'm supposed to be making twice my salary. Goddamnit!

See The Market for Lemons [wikipedia.org]. The existence of tons of bad programmers, and the inability of employers to tell them apart from good ones, drives the salaries of all programmers towards that of the average programmer.

Re:Sigh.

Ha! Trust me on this, COBOL programmers will take their goddamn jobs to the grave with them. Those S.O.Bs get called out of retirement and paid consultant money to fix things they arsed up decades before.

Anything that sits on a financial system will be changed as seldom as possible, and COBOL is alive and well with people who maintain those damn ancient not-to-be-upgraded systems.

Re:Ant farm engineer

Your wish is google's command:

http://dribibu.xs4all.nl/dilbert19950813.html [xs4all.nl]
http://dribibu.xs4all.nl/dilbert19951230.html [xs4all.nl]

mythical man month

that is all.

They exist, but they don't know it.

Here's what Paul Graham had to say about Great Hackers:

Because you can't tell a great hacker except by working with him, hackers themselves can't tell how good they are. This is true to a degree in most fields. I've found that people who are great at something are not so much convinced of their own greatness as mystified at why everyone else seems so incompetent.

http://www.paulgraham.com/gh.html [paulgraham.com]

More to it than skillz

There are other issues besides technical skills. The higher you rise in the food chain, the more the "soft skills" matter. Organizational skills, people skills, communication skills. All the elegant code in the world doesn't make up for a prima donna who won't show up for a critical meeting or who openly disrespects "lesser" members of the team. The last thing in the world most people want is to hire the developer equivalent of Terrel Owens... because, just like Owens, they will leave damaged teams in their wake. Morale counts. The reason that leads get paid more than individual contributors is not just because of technical skills. It's because they can herd cats. It's because they can recognize that business reality sometimes has to trump "ideal" elegance or philosophy-of-the-week. It's because they can convince Dev to talk to QA to talk to Product Management to talk to Sales. It's because they can somehow get a clear functional spec from the marketing guy. It's because they can get by with existing equipment instead of demanding an Intel Core 31337 for their desktop. It's because they don't have to have an HR apologist in tow smoothing ruffled feathers everywhere they go. "Senior" implies so much more than "technical guru".

Re:Internal Inconsistency in his Argument

I worked for a company that got bought by a bigger company.
We had an über programmer. He left because rather than exceptional pay, he wanted good enough pay and a small company style work life.

We then got in a pickle where some kernel mode drivers for NT4 needed to be revised and SoftICD'd in. Even though he doc'd everything and gave training to our programming staff about gotchas and pitfalls as well as maintenance, it was something that only about 100 people in the world could really do. All we could get done is a widely variating series of BSODs. We hired him back at $12K/day + travel for 5 days of work. He did work his ass off, further document everything, and provide additional spot training to our two brightest. The job was done and he had a check for $60K. I suspect that the training and docs took the vast majority of his time. I asked him why so much (and why MegaCorp would pay that) and he said it was simple. They were a big company not interested in paying him either in lifestyle changes or money so he didn't stay, but for a short job he charged what it was worth.

The free market did work. (considering his solution was cheaper than a contract with MS for the same work by $40K).
-nB

Re:Internal Inconsistency in his Argument

But I think that if the über-programmer really does exist, then eventually the free market will figure that out, and compensate him accordingly.

It has, and then some. These "über-programmers" are what you and I know as "wildly successful startup founders." Part of the reason it's so hard to hire them is because they are mostly already independently wealthy and / or personally invested in a project of love that no offer of cash and benefits will draw them away from. Most likely, if the former is not true, the latter will eventually cause it to be true. The best and most common way of hiring an über-programmer is to buy the company they currently work for.

Re:Internal Inconsistency in his Argument

Not really. The uber programmers rarely have the skill set needed to found a buisness. Those tend to require high people skills and financial skills, which are almost completely disjoint from the programming ones. Plus it would mean spending time doing all that bullshit, instead of programming.

The real answer is that most uber programmers work for that small premium. They get to do what they want, get a few other perks like their choice of assignments, and are generally happy as is. Money isn't really the key motivator for them- if it was, they'd be in another field that pays more.

Re:Internal Inconsistency in his Argument

There's no one programmer who does the work of ten other programmers. One uber-programmer does just as much work as one ordinary programmer. It's just that the results solve ten times as many problems. Programming is fundamentally a design problem. A great bridge designer doesn't do the work of ten lousy bridge designers; the great one designs one great bridge in the time it takes the ten lousy ones to design ten lousy bridges.

The best approximation is that each problem has a certain complexity and a certain size. The size determines how long it will take, and it doesn't matter how good the developers are. The complexity determines how good a developer is needed to make progress at all. If you've got only easy problems, an uber-programmer doesn't help you much (unless the programmer can find a smaller, harder problem that replaces the big easy one). If you've got a hard problem, ten average programmers will work on it forever without getting any results.

And there's one last thing specific to computers: the computer can solve easy problems for you, but making it do so is a hard problem. But solving that one hard problem (plus some processor time) resolves a lot of easy problems. Another type of hard problem is writing a magic library function that makes a range of moderately hard problems easy enough for average programmers to solve.

If you've got ten people essentially doing data entry, an uber-programmer may be able to eliminate the need for them to do that at all. If you've got ten developers working on some problem, an uber-programmer may be able to double their productivity. In either of these cases, the uber-programmer directly produces something that isn't part of the actual project, but the benefit to the project is on the order of ten average programmers' work. And, if the uber-programmer reduces the complexity of the problem to put it in reach of the rest of the team, no amount of ordinary programmers' work would benefit the project as much as the uber-programmer's contribution. Of course, if you require an uber-programmer to literally do the work of average programmers, there's no benefit at all.

ueber programmers do exist!

>There's no one programmer who does the work of ten other programmers

Oh there is, I am one. I've had several occasions where I solved problems in a few days (or 4 hours once) that others were struggling at for months; some of these others were plain incompetent, others were pretty good.

I find that I'm pretty unique in fitting the right tool to the problem at hand, as well as in general overview. I've never met anyone as productive as myself.

I'm posting this anonymously, because I have to work with others, and one of the things you cannot do is alienate everyone around you; one sure way of doing that is being more skilled than them in all job related aspects.

Best damn article in a while

While I am not a dev, Sysadmin here, this is probably the best article I have read on the subject in a long time. This idea of lets get someone in and train them up is assinine. Of course not every company can afford 120K a year but what about the lower end, midwest people get hit up with 45K a year jobs all the time, if the company would jump to 60-70K they would get 2X the dev and also get a much better product. I am currently with a company that made the mistake of hiring a below par employee to dev a site. Now they lucked out and got someone for the same price who doesnt care about salary but it a hell of a PHP developer, probably the best I have ever worked with. He spends 90% of his time fixing mistakes of the last dev and does things in minutes that took his predecessor days.

Same concept goes with my job field, I spend a considerable amount of time consulting, fixing poorly configured networks and servers. You cant just grab a joe off the street and expect him to be a professional or put out professional work without having learned his/her lessons, they will make mistakes learning, do you want it to be on your buck and your network?

Re:Best damn article in a while

This idea of lets get someone in and train them up is assinine.

Dumb question, but if nobody trains new developers, then where the heck are those more experienced developers supposed to come from? And of course the related question, where did the few that we now have come from?

T

Re:Best damn article in a while

Some people are self-taught or learn on the job. Not everybody needs to learn from another person.

Re:Best damn article in a while

welcome to the working world. The specialized knowledge you seek doesnt come from your first job. It comes from taking a thousand crap jobs and compiling information as you go. If you seek to be specialized in mainframe development then you have a long path in front of you, as there are many more experienced mainframe developers with a good 15 years of experience ahead of you.

However if you are one of the aforementioned mainframe experts then you need to take a crap job or some side jobs as a web developer, starting off working for free for non-profits and the like, then as your skills progress you can start charging then work your way up from there. Then you can start looking for the mainframe web devs that you are aiming for. However if you are like most people you become a specialist because that is where the job field led you. You may see that there is a good paycheck in the area or it may interest you but that is exactly why those jobs pay well, because they are a rare person and unless a trade school offers a program you are SOL.

The purpose of the article is to explain that a good dev can make up for some serious problems as a beginner programmer, or if your company is big enough you can hire the underlings to do the crap coding while the expert does all the engineering, planning and actual Development. Those entry level people pay their dues coding and recoding the same scripts over and over for the guru and after a while they move up, taking that knowledge with them. However if a company cannot afford a whole host of devs, they had better hire the good ones rather than the entry level employees.

Re:Languages

The hardest part of finding a new job is getting past the recruiters, who are generally not capable of anything more than keyword matching against your experience. Use your contacts if you can to get in front of the technical managers who will understand that your domain knowledge and overall experience is more valuable than which languages you have been using for the past few years.

Re:close your browser now boss

I'm new to it all also. I've worked at this company a few months and have designed two separate applications, while maintaining/improving a third. All I can say is it seems that I'm a better developer than the people that came before.. The project I've had to maintain is garbage spaghetti code that was said to have been developed over a few years, and would take me 6 months tops to write from scratch. I'm not saying I'm good at what I do but I am saying whoever came before was terrible at what they do. This boggles my mind considering that it was written by 'Senior' developers, who probably make double what I do.

I obviously haven't seen the code you're talking about and hove no opinion of it. However, I've heard this crap from new programmers for years. So let me through some ideas your way.

1. it isn't garbage and spaghetti because you have difficulty following the technique, there are a thousand ways to do anything. It's garbage and spaghetti when it doesn't work well, and when it's under documented.
2. Could you really write it in six months without referencing the code you are talking about? It's one thing to write code when the problems have been solved, quite another to solve the problems.
3. the scope of a job changes over time. For a new programmer, start looking for scope creep, it's a friend and an enemy.
4. code changes over time because languages change over time. Look for those situations in the spaghetti code where those guys that you are better than wrote functions that were not available in the language. You may get a surprise that they did something really elegant to overcome something missing in the language originally that exists now.
5. lets say that there were five programmers on the code before you, one after another. The first four might have been gods gift to programming, it only takes one pretentious newbie in the chain to really screw up pretty code.

certainly none of these apply to your project, but it's something to think about as you are exposed to projects in your career.