How Facebook Ships Code 314
Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."
no process (Score:5, Funny)
That explains a LOT...
Re:no process (Score:5, Interesting)
It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200
So basically, Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying, where the worst thing someone can possibly do is suggest an idea that an engineer claims is "stupid" and doesn't give them opportunity for nerd glory. How much you want to bet that Zuckerberg sets the tone and decides that any modifications to the way the privacy settings are run is "stupid" and "boring."
Re: (Score:3, Interesting)
Like Google, unlike Apple.
I don't mind a professional engineer for a boss, but any which displays the traits celebrated by xkcd would be absolutely unbearable.
Re: (Score:2)
So you'd prefer your fictional engineers from Dilbert?
Re:no process (Score:4, Insightful)
The engineers in Dilbert are fine, it's the managers you don't want.
Re: (Score:2)
I would pass on Wally...
Re: (Score:3)
I AM Wally, you insensitive clod!
Re: (Score:3)
I would pass on Wally...
There are consistent references that Wally was a great engineer in his youth. Years of bad management has disillusioned him to the point where he realizes nothing he does really matters, so he might as well do nothing.
Basically, Wally is what Dilbert would end up being with 30 years of experience.
Re:no process (Score:4, Interesting)
Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying
At any other company, that could be counter-productive. But considering that the tool they're building is basically the ideal tool for building your clique and growing your influence, it probably just adds to their "dogfooding."
The Cheetah and the Elephant (Score:4, Interesting)
At a staff of 200, I would also wager that new features take a back seat and would propose that this is why we see Facebook's "upgrades" as being almost purely cosmetic (i.e. layout, markup, etc.).
Re:no process (Score:5, Insightful)
I'm always curious about this expression, which - just like "I could care less" in place of "I could not care less" - is the opposite of what's meant. Surely, 200 would be astonishing, let alone 500.
Re:no process (Score:4, Insightful)
Re:no process (Score:4, Insightful)
Which is exactly my point. It's amazing that the casual way they handle code could work with 200 programmers, let alone 500. See? "Let alone" means "never mind." As in, it's hard enough for 200, and 500 isn't even worth mentioning
Re:no process (Score:5, Insightful)
Re: (Score:3)
Oh the huge manatee?
Re:no process (Score:5, Funny)
Re: (Score:3)
Not a contraction, since the omission is not internal. More likely it was first used as a sarcastic play on the original phrase, and then the tone was dropped by imitators who understood the form but not the function.
Re: (Score:3)
It also sounds like all changes are controlled by comparing them against some sort of metric when distributed to t
Re:no process (Score:4, Interesting)
Something TFA doesn't talk about is how the engineers are organized -- surely there's a team that does memcache, a team that does database work (like those Cassandra guys), a team that does i18n, a team that does rendering, a team that specializes in Flash doodads, etc. It's hard to see how the engineers check each other without knowing where demarcations lie.
My issue wouldn't be with the quality of the code, it's how they decide what code to write in the first place. All of the Kaizen processes work great on an assembly line, but a feature request and a coworker evaluation aren't fender panels. They require someone in authority to tell you what you're trying to accomplish in the first place. The description of the way people decide what to add (basically shutting out the marketing people if they speak out too much) and how they discipline each other (too many blames in the SVN log and too many "public shamings" result in termination) speaks to a culture where new ideas and customer focus are almost stigmatized, and where the engineers don't try to evaluate each others ideas as much as they worry about getting voted off the island.
Re: (Score:3)
That's a 'wholly-owned subsidiary' so that there's no conflict of interest. *wink*
Oh, wait, we're past the point where that sort of thing is hidden. Yup, forgot the data mining team...
Re: (Score:3)
It is astonishing. I guess that's why nobody was doing it before, and they were able to get rich from a mere website in what seemed to be an already established niche.
Most of the comments here are looking at the process and predicting a result. That's backwards. We can see the result; the company is extremely successful. If that doesn't match your mental model of processes, maybe the mental model needs updati
Re: (Score:3)
Most of the comments here are looking at the process and predicting a result. That's backwards. We can see the result; the company is extremely successful. If that doesn't match your mental model of processes, maybe the mental model needs updating.
Even if the company is (arguably) successful, this still may be despite, rather than thanks to, their processes.
Re:no process (Score:4, Insightful)
based not on consensus or managerial decision making but by cliques and lobbying
I have worked, via contract, in at least a dozen medium to large sizes companies where this is the underlying mechanism. If you stick out in that you focus solely on the work and do not engage with the de facto clique leaders, who as often discuss droll subjects such as retiling their bathroom as often as they collectively decide how they want you to do their jobs for them, then absurd personal complaints will begin to stack up against you and you will be forced out of a job.
I used to flaunt my individuity, but since 9/11 I have been consciously conforming to old school business formality... never arrive late, never leave early, never get sick or take off, always dress nice, keep short hair, no superfluous non-work related conversations, never mentioning anything from my personal life, always ready with a polite smile and an enthusiasm for the work no matter how mundane it is... I've tried to embody what the ideal is for a person in my field. This doesn't work. One needs to become *good* friends with whomever that defacto leader is, who is as often as not NOT the management (but, indeed, through strength of personality has the manager under their thumb, and completely snowjobs the executives with an almost supernatural confidence).
If they like you, you win. And when you win, they like you. Otherwise... you are the first of any approaching cutbacks... you become the sacrifice that saves everyone elses jobs, the scapegoat for any complaints that come down from the executive levels.
The last 2 teams I worked with were more like a gang than a corporate division, whose self-preservation far outweights the work they are responsible for (My last contract ended abruptly when 3 of my counterparts in another building took heavy criticism for their laziness and distict lack of any work ethic... but because they were well liked, somehow I got the boot and the blame for their incompetance... without even remotely having anything to do with whatever incident occurred (never quite clear on exactly what it was... something to do with an assigned task that they kept passing between them and was never completed).
I hope someday my perserverance will pay off and I will be able to work not only with smart people (intelligence was rarely an issue), but educated and honest, and hopefully enlightened individuals that do not synthesize drama to manipulate perceptions such that those they like, towing the gang's loyalty, remain employed, while those that are effective, and thus throwing the curve, are terminated.
Engineers? (Score:2, Insightful)
I'm just laughing (sadly) that web developers are considered engineers.
But hey, they call garbage collectors "Sanitation engineers" so why not Javascript and PHP coders...
Re: (Score:2)
I've never heard anyone call a garbage collector a sanitation engineer, though I have heard them called sanitation workers.
Re: (Score:2)
Engineer is defined as: A person who designs, builds, or maintains engines, machines, or public works.
A web developer is someone who designs, builds, or maintains software.
I suppose you could argue that software is not an engine, machine, or public work. However, the first half of the definition is quite applicable.
Re: (Score:2)
Wait, the language you write code in dictates whether or not you're an engineer?
And here I thought that engineering was the application of science and math to the creation of practical solutions for societal problems and needs.
But I guess what I learned is wrong, and "engineer" comes from the Latin root word that means "knows how to program in C++".
Re:no process (Score:5, Funny)
Facebook does have a formal QA group, it is called the user base.
Re:no process (Score:4, Funny)
That explains a LOT...
Personal journal: This company trusts me with the code base. I don't know why. Dumb fucks!
Re: (Score:2)
Jump in it!
Facebook ships code? (Score:3)
Then where can I get that shipped code?
Although, after reading that story, I don't think I want that code. It sounds like a textbook example of feature and focus creep.
Re: (Score:2)
It sounds like a textbook example of feature and focus creep.
From using facebook it seems like they lack a real plan anyway. When they're not busy ripping of the hot new thing, feature creep is their roadmap.
Why would Facebook need 500 engineers? (Score:2, Funny)
It's not like anything really changes on it.
You don't get Facebook (Score:2)
How would you know? Are you one of their paying customers?
Re: (Score:2)
I was wondering that myself.
How many engineers does Apple have working on OS X? Or Google on Android?
I don't see 500 engineers worth of results on FaceBook.
Re: (Score:3)
Have you handled 300 million users lately? Just the backend storage scaling itself probably eats a hundred engineers... then there's the code performance engineering, php compiler and memcached wranglers... there's another 100 engineers. And of course there's the network engineers! And the Apache engineers(50) and the cable engineers, the customer support engineers... OH and the social engineers!!
That leaves about... (click click click...) 4 engineers to work on the interface code.
Re: (Score:3)
Heck, do you need to invent new ways to harvest data, when after your 4-6 weeks you get unfettered access to the entire database? Seems like a security/privacy nightmare (for us, anyway - FB doesn't seem to care.)
as a nerd (Score:2)
i share the worry facebook represents in terms of privacy and the future of the internet with everyone else here
but at the same time, i get a warm feeling in my pants thinking about the programming and workings of an enterprise this big and powerful
and if you laugh at me for getting that warm feeling in my pants, you're clearly on the wrong website
Re: (Score:2)
and if you laugh at me for getting that warm feeling in my pants, you're clearly on the wrong website
You are on the wrong web site. The nerd doesn't care about "an enterprise [...] big and powerful".
Re: (Score:2)
you're talking about the emo nerd, a subset of nerddom, not representational of all nerds
some nerds are more of the evil scientist variety. and while sharks with friggin laser beams on their heads are more their kind of psychosexual gratification, facebook is not bad either in terms of pants warming evil plots
Re: (Score:2)
Shuffleboard programming at 10:00 in the Big Room!
No access controls? (Score:5, Insightful)
After boot camp, all engineers get access to live DB
So anyone who's ever worked at FB as an engineer will have likely downloaded copies of all their friends' / family's / ex-girlfriends' inboxes, chat history, etc.
Not surprising really.
Re: (Score:3)
I could've sworn there was an article on /. some time ago that indicated Facebook employees had to justify accessing "personally interesting" information they didn't have legitimate access to through their FB profile, namely, the sorts of things you mentioned: inboxes of exes, family members, etc. There was a master password and all uses of it were recorded and audited so you had to explain yourself whenever you used it. My memory might be a little fuzzy but it seemed that FB employees didn't have unmonitor
Re: (Score:3)
Re: (Score:2)
No access controls doesn't mean no oversight.
Re: (Score:2)
Um, yes, it does. Access control is not just denial, it's also logging. If you don't have any idea who's accessing what, you can't possibly have any oversight.
Re: (Score:2)
Permissions are distinctly different from logs. Both are "controls" in the audit sense of the word, but this is not a case of access control.
Re: (Score:2)
What you thought you had privacy on facebook?
Your best bet is to be uninteresting, which for the /. crowd is quiet easy.
Programmers != Engineers (Score:2, Insightful)
Once and for all, programmers are not Engineers.
Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.
Think about it.
If facebook was staffed by real engineers, a privacy breach would cost the engineer his/her license, and they could be personally sued, as well.
But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up m
Re: (Score:2)
Careful now. Some of the dot-bomb programmers have managed to hang onto their employment and they regard their wage as proof of competence.
Re:Programmers != Engineers (Score:5, Informative)
Not all engineers are licensed. Civil engineers are usually licensed. Mechanical engineers and electrical engineers are usually not licensed. Similarly, there is no licensure for system engineers. There are "certifications" but these are essentially meaningless.
Re: (Score:2)
Re: (Score:3, Informative)
I have a degree from a university in electrical engineering. I work as an electrical engineer and I consider myself one. I am not licensed as a professional engineer (PE) and have little interest in obtaining that license at this point as the type of work simply doesn't appeal to me.
As you suggest, those PEs do put their license on the line when they sign a document. From my understanding, companies that have resident PEs will only have a few and have other non-licensed engineers do the less expensive wo
Re: (Score:2)
Lots of people call themselves engineers. The meaning of the word is changing.
I think Professional Engineer might imply licensing. In fields like software where licensing and insurance isn't even an option, nobody could say they were mislead by the title of "Software Engineer".
Re: (Score:2, Funny)
But then again, anyone in the good old US of A is an engineer. From the sanitation engineer who picks up my trash, to the beverage refreshment engineer who makes my coffee at Starbucks.
Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.
As one of slashdot's many Comment Moderation Engineers, I believe you bring up a good point - henceforth Anonymous Coward shall be known as Anonymous Engineer.
Re: (Score:2)
And the industry likes it that way.
Why? it keeps programmers as a dime-a-dozen level. If we were actually real Software Engineers, than we would be getting engineer pay, the managers cant force us to ship it if it compiles as engineering allows you to tell management to STFU if it's unsafe.
Corporations worked hard to make sure that programming and IT would be a factory level job.
Re: (Score:2, Insightful)
Ya well I knew a "DBA" that was responsible for a MS Access DB. Not actually even creating it... just running it. Yes one. Couldn't even script, hired consultants for that. Seriously. Made more money than me.
Also I have known people that called themselves CEO or CFO, etc... but were basically a one man show working from home.
Titles are cheap. I always wanted to put "Supreme Overlord" as a title on a resume just as a joke. It has about as much meaning as some of the crap I have seen other people use.
Re:Programmers != Engineers (Score:5, Informative)
No, this is not true. There is a big difference between being the employee of a company and being an officer of the company. Those "O" titles actually mean something. Doesn't mean that the people who are officers of the company are the right people for those roles, but there's real baggage that comes with those titles, including a higher standard for the consequences of entering into contracts, obligating the company to act or pay bills, etc.Being an "O" also makes you more of a law suit magnet.
Re: (Score:2)
Licensed engineers != Engineers (Score:3)
Nowhere does this mention licensing of any kind. Licenses and liability to lawsuits are a recent
Re: (Score:3)
Okay, thought exercise:
You're walking down the street, and somebody walks up with a knife and demands your wallet. In the ensuing scuffle, you end up stabbed 3 times in the chest. Nothing instantly fatal, but your lungs are now filling with blood.
A passer-by calls an ambulance for you, and once you're loaded in and ready to go, the EMT pulls out a list of all the people who practice medicine in the area and says, "which doctor do you want to have care for you?"
You weakly point at one of the names on the l
Re: (Score:2)
Engineer is just another cheap title, like CEO/CFO/CIO/CTO, etc, free to be used by anyone.
I resent that remark! My lawyers will be in touch!
Cordially yours,
discord23
Chieft Toilet Officer
Executive Sanitation Engineer
Craptastic Inc
Re:Programmers != Engineers (Score:5, Insightful)
Engineer is defined as someone who designs, builds, and maintains.
It has nothing to do with licensing, government accountability, code of ethics, accreditation, or anything else.
It is not a cheap title, it is a description of the job. When they say Facebook engineer, we get a sense of what those employees are required to do; which extends beyond the role of just programming.
Re: (Score:3)
In the narrow sense of holding a P.E. (Professional Engineering) license or equivalent, no. But the P.E. does not define the engineer, no matter how many holders of such licenses think it does.
Re: (Score:3)
Once and for all, programmers are not Engineers.
Programmers can not be held liable, have no professional license, no government accountable body, no code of ethics, no liability insurance, or other measures in place to directly protect the public.
Think about it.
Professional Engineer != Engineer
Unless of course you wish to make the argument that making say, a space shuttle, doesn't require any "real" engineers or didn't you know that aerospace engineering is largely unregulated with no professional license, no government accountable body, no code of ethics, no liability insurance or other measures in place to directly protect the public from the individual engineer rather than the company or organization? The same thing with the majority of mechanical engineers, a
Re: (Score:2)
"Pretty much common sense" means that a lot of it is a vague set of "principles" that members of the ACM are expected to abide by, and which many of them ignore when it doesn't match their interpretation of "common sense." And let's be quite honest - there's probably about... five... people on Slashdot who feel themselves bound by section 1.5 of the ACM COE:
Fake (Score:5, Informative)
Re:Fake (Score:4, Interesting)
Great, a series of statements about Facebook by some people not speaking on behalf of Facebook corrected by one person claiming to be from Facebook but not speaking on behalf of Facebook. Who knew Facebook engineers and fanboys were as capricious and unwilling to understand boundaries as the users?
Re: (Score:3, Interesting)
Sir the post you cite actually confirms a lot of this is true. Such as the lack of QA. They are all encouraged to test and then report any bug in a new internal version? Is that supposed to be sufficient?
Just more proof that Facebook really doesn't care about anything except driving eyeballs to ads. I bet _that_ system is NASA engineered.
Re: (Score:2)
what this means is... (Score:2, Interesting)
...that they have very poor/disorganised engineering practice, and they're trying to sell a bug in their methodology as a feature. It's like when Google suddenly fluffs something up with your search results momentarily, and you think "huh, that's weird," but actually they're just testing (or think they're testing... in reality, they're just annoying you).
Since there's nothing technically interesting or requiring resilience about anything Facebook has to offer, they can get away with it.
Re: (Score:2)
What's the name of the fallacy in which secunda requires prima to reveal that prima is Gandhi before prima may criticise Mussolini?
down the road (Score:2)
Re: (Score:2)
Re: (Score:2)
Could this explain why Goldman do not want to have Facebook stocks in the US.
Re: (Score:3)
Re: (Score:2)
Intentionally malicious code, eh? (Score:3, Insightful)
But will they catch unintentionally malicious behavior that gets un-reviewed code in?
It's a component system (Score:4, Interesting)
Bear in mind how Facebook works internally. It's a large number of programs intercommunicating through a remote procedure call system. There's no one big "build". The interfaces between programs are well defined, and changing out programs individually is normal.
"dogfooding"? (Score:4, Insightful)
Yeah, I had to look it up. Never heard of the term "dogfooding" before.
It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".
Re:"dogfooding"? (Score:5, Funny)
The choosing is weirding, you're right, but as long as the meaning is clearing, it's not really probleming.
Re: (Score:2)
Yeah, I had to look it up. Never heard of the term "dogfooding" before.
It seems a silly word choice, since in context it carries no more meaning than the rather more conventional word "using".
Well, it could be worse...
"In 2009, the new CIO of Microsoft, Tony Scott, argued that the phrase "dogfooding" was unappealing and should be replaced by "icecreaming", with the aim of developing products as "ice cream that our customers want to consume"
Icecreaming, you screaming...
Re: (Score:2)
That said, assuming that you are a software engineer, you really should have come across that term before, it's been around for 20 years, and in common use for at least ten.
Re:"dogfooding"? (Score:5, Informative)
So how then...... (Score:3)
Does idiotic ideas like sharing my private info with any random app that is easily setup to be malicious get through?
Why dont they have a checks and balances with the executives that should be smacked hared in the head for their bad ideas?
Same old tune (Score:5, Insightful)
We get to read this story every few years. Some company makes a popular product and all of a sudden they are the authority on development. News flash guys, enterprise software wasn't just invented and we've been rolling out features to millions of people for years. In environments where any discrepancy can lead to serious legal repercussions. So just for a second, hear us out. We know what we're doing and we don't need to reinvent it every few years because the guys from Facebook, Basecamp, Hacker news, or any other place said we're doing it wrong. Our bottom line and years of quality deliveries say we're doing it right. What do these guys have besides a history of rolling out mistakes? Their value is based on hype, not their code quality. Some of us don't have the luxury to make huge mistakes and have our customers be okay with it.
Re: (Score:3)
This explains bizarre feature regressions (Score:5, Interesting)
I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.
A few:
- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*
- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.
- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.
- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Status updates are effectively gone. They are no longer stuck to the top of your profile and get buried in your own news feed. Sure, they still show up in the main news feed where it may or may not be seen thanks to the amount of info being dumped on the page, but it seems pretty nonsensical to have someone go to your profile and not be able to see your status at a glance.
As for photos, let me break this down.
When...
Of course they have no QA... (Score:4, Interesting)
It's easy not to have QA when your users aren't paying you.
Google's in the same boat - the websites aren't the product, the end users are. When your website is your product this crap won't fly.
"Compnay" (Score:2)
I guess slashdot is also missing a bit of QA :)
Insider Attacks (Score:3)
Urgh. That means every competent insider can easily put disguised backdoors into the code to be used later. Ordinarily you want to severely limit who can do that. In fact one fundamental principle when operating high security applications is that development personnel must not have access to the production environment, exactly to not hand all eggs to everybody.
Obviously a young and highly risk-taking company. Not good at all.
Re: (Score:2)
Did he actually do this? I've read a paper he wrote about this, but got the impression it was a proof of concept for hiding a backdoor in the compiler, not something that actually shipped.
If I'm wrong I'd be interested to read more.
Re: (Score:2)
Re: (Score:3)
Scaling a site to be the 2nd most popular site on the web is no small feat. I would imagine a good number of people on their tech team are designated to managing the load.
Re: (Score:2)
I seriously have my doubts that 3-4 engineers could completely redesign the website every six months on top of regular maintenance.
Very common sentiment (Score:4, Informative)
> I'd have thought that a team of 3-4 engineers could achieve the same effect.
This is very typical sentiment in people with little to no experience in running large sites. They believe a) What they see is all the exist, b) Scale does not matter, and c) The site never changes.
This lead them to the faulty conclusion that any high-availability, high-traffic site could be run by two people (of which they no doubt are one) live-updating PHP scripts on the fly. See Also: Dunning-Kruger effect [wikipedia.org].
Re: (Score:3)
http://wow.joystiq.com/photos/adgc09-the-universe-behind-world-of-warcraft/#2296009 [joystiq.com]