How Facebook Ships Code

Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."

No access controls?

[choongiri]

After boot camp, all engineers get access to live DB

So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.

Not surprising really.

Programmers != Engineers

[Anonymous Coward]

Once and for all, programmers are not Engineers.

Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.

Think about it.

If facebook was staffed by real engineers, a privacy breach would cost the engineer his/her license, and they could be personally sued, as well.

But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up my trash, to the beverage refreshment engineer who makes my coffee at Starbucks.

Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.

Tired of it all

[Anonymous Coward]

Way too many Facebook stories today. I wish that website would disappear. Just walk anywhere in public for 15 minutes and count how many times you hear a reference to this website. Sometimes the count reaches double digits. It is ridiculous. You would think there was nothing else in the world that mattered. People are boring *yawn*

Intentionally malicious code, eh?

[noidentity]

It would take intentionally malicious behavior to get un-reviewed code in.

But will they catch unintentionally malicious behavior that gets un-reviewed code in?

"dogfooding"?

[Tetsujin]

Yeah, I had to look it up. Never heard of the term "dogfooding" before.

It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".

Engineers?

[Anonymous Coward]

I'm just laughing (sadly) that web developers are considered engineers.

But hey, they call garbage collectors "Sanitation engineers" so why not Javascript and PHP coders...

Same old tune

[trollertron3000]

We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.

Re:Programmers != Engineers

[DarthVain]

Ya well I knew a "DBA" that was responsible for a MS Access DB. Not actually even creating it... just running it. Yes one. Couldn't even script, hired consultants for that. Seriously. Made more money than me.

Also I have known people that called themselves CEO or CFO, etc... but were basically a one man show working from home.

Titles are cheap. I always wanted to put "Supreme Overlord" as a title on a resume just as a joke. It has about as much meaning as some of the crap I have seen other people use.

Re:no process

[ScentCone]

It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200

I'm always curious about this expression, which - just like "I could care less" in place of "I could not care less" - is the opposite of what's meant. Surely, 200 would be astonishing, let alone 500.

Re:no process

[somersault]

The engineers in Dilbert are fine, it's the managers you don't want.

Re:Programmers != Engineers

[mini me]

Engineer is defined as someone who designs, builds, and maintains.

It has nothing to do with licensing, government accountability, code of ethics, accreditation, or anything else.

It is not a cheap title, it is a description of the job. When they say Facebook engineer, we get a sense of what those employees are required to do; which extends beyond the role of just programming.

Re:no process

[tsm_sf]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Re:no process

[ScentCone]

Except that tasks become increasingly difficult the more nerds you throw at them, not less. 500 engineers makes a project harder than 200.

Which is exactly my point. It's amazing that the casual way they handle code could work with 200 programmers, let alone 500. See? "Let alone" means "never mind." As in, it's hard enough for 200, and 500 isn't even worth mentioning ... so, let it alone. And that's why I pointed out that the phrase was being used exactly backwards.

Re:no process

[catmistake]

based not on consensus or managerial decision making but by cliques and lobbying

I have worked, via contract, in at least a dozen medium to large sizes companies where this is the underlying mechanism. If you stick out in that you focus solely on the work and do not engage with the de facto clique leaders, who as often discuss droll subjects such as retiling their bathroom as often as they collectively decide how they want you to do their jobs for them, then absurd personal complaints will begin to stack up against you and you will be forced out of a job.

I used to flaunt my individuity, but since 9/11 I have been consciously conforming to old school business formality... never arrive late, never leave early, never get sick or take off, always dress nice, keep short hair, no superfluous non-work related conversations, never mentioning anything from my personal life, always ready with a polite smile and an enthusiasm for the work no matter how mundane it is... I've tried to embody what the ideal is for a person in my field. This doesn't work. One needs to become *good* friends with whomever that defacto leader is, who is as often as not NOT the management (but, indeed, through strength of personality has the manager under their thumb, and completely snowjobs the executives with an almost supernatural confidence).

If they like you, you win. And when you win, they like you. Otherwise... you are the first of any approaching cutbacks... you become the sacrifice that saves everyone elses jobs, the scapegoat for any complaints that come down from the executive levels.

The last 2 teams I worked with were more like a gang than a corporate division, whose self-preservation far outweights the work they are responsible for (My last contract ended abruptly when 3 of my counterparts in another building took heavy criticism for their laziness and distict lack of any work ethic... but because they were well liked, somehow I got the boot and the blame for their incompetance... without even remotely having anything to do with whatever incident occurred (never quite clear on exactly what it was... something to do with an assigned task that they kept passing between them and was never completed).

I hope someday my perserverance will pay off and I will be able to work not only with smart people (intelligence was rarely an issue), but educated and honest, and hopefully enlightened individuals that do not synthesize drama to manipulate perceptions such that those they like, towing the gang's loyalty, remain employed, while those that are effective, and thus throwing the curve, are terminated.

Re:no process

[NoSig]

It doesn't just sound silly, it's a crime against intelligence is what it is. I doubt it's a contraction like you say since "as if I could care less" is not something I've ever heard, while "I couldn't care less" is. It's just a bunch of people who don't know what they are saying that cut out the "not" because they don't understand how the sentence is supposed to work. That's like contracting "I don't like Hitler" to "I like Hitler". I guess by your reckoning it would be like contracting "as if I like Hitler" to "I like Hitler" which is just as stupid, but I still think it's the former.