How Facebook Ships Code 314
Hugh Pickens writes "The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week 'Boot Camp' training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to the live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. 'All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to,' writes yeegay. 'It would take intentionally malicious behavior to get un-reviewed code in.' What is interesting for a company this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day."
what this means is... (Score:2, Interesting)
...that they have very poor/disorganised engineering practice, and they're trying to sell a bug in their methodology as a feature. It's like when Google suddenly fluffs something up with your search results momentarily, and you think "huh, that's weird," but actually they're just testing (or think they're testing... in reality, they're just annoying you).
Since there's nothing technically interesting or requiring resilience about anything Facebook has to offer, they can get away with it.
Re:Fake (Score:4, Interesting)
Great, a series of statements about Facebook by some people not speaking on behalf of Facebook corrected by one person claiming to be from Facebook but not speaking on behalf of Facebook. Who knew Facebook engineers and fanboys were as capricious and unwilling to understand boundaries as the users?
Re:no process (Score:5, Interesting)
It's astonishing that they can keep such a process rolling with 500 engineers, let alone 200
So basically, Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying, where the worst thing someone can possibly do is suggest an idea that an engineer claims is "stupid" and doesn't give them opportunity for nerd glory. How much you want to bet that Zuckerberg sets the tone and decides that any modifications to the way the privacy settings are run is "stupid" and "boring."
Re:no process (Score:3, Interesting)
Like Google, unlike Apple.
I don't mind a professional engineer for a boss, but any which displays the traits celebrated by xkcd would be absolutely unbearable.
It's a component system (Score:4, Interesting)
Bear in mind how Facebook works internally. It's a large number of programs intercommunicating through a remote procedure call system. There's no one big "build". The interfaces between programs are well defined, and changing out programs individually is normal.
Re:no process (Score:4, Interesting)
Facebook is run by an aggressive engineering culture based not on consensus or managerial decision making but by cliques and lobbying
At any other company, that could be counter-productive. But considering that the tool they're building is basically the ideal tool for building your clique and growing your influence, it probably just adds to their "dogfooding."
This explains bizarre feature regressions (Score:5, Interesting)
I know they're always looking to "streamline" Facebook, but sometimes really obviously good features vanish without explanation and leave users scratching their heads. These decisions would make more sense if you could see some way that FB was making life easier for advertisers or something, but often, the feature regressions are just nonsensical to the extreme.
A few:
- Facebook got rid of statuses. The one e-crack feature they're best known for. Gone with the new profile. Now they're just wall posts to yourself that quickly fall down the page. *facepalm*
- Photos are now uploaded in descending order. It used to be that a group of images would be uploaded like this: beachtrip-1.jpg, beachtrip-2.jpg... and so on. Now it reverses the order of an uploaded album. The last photos taken are displayed first. Posting vacation pics? Well your friends get to see the day you left and work their way back to the day you arrived.
- The "Reverse Photo Order" option was removed when the above "feature" was introduced. You can drag around photos to manually reorder them, but every photo that you mouse-over jumps out of the way and moves to different rows. It's a UI disaster.
- A few months ago I started seeing tagged photos of friends despite the fact they were in private albums of people I'm not friends with who also happen to have locked-down profiles where you can't see anything at all unless you add them.
Re:Fake (Score:3, Interesting)
Sir the post you cite actually confirms a lot of this is true. Such as the lack of QA. They are all encouraged to test and then report any bug in a new internal version? Is that supposed to be sufficient?
Just more proof that Facebook really doesn't care about anything except driving eyeballs to ads. I bet _that_ system is NASA engineered.
The Cheetah and the Elephant (Score:4, Interesting)
At a staff of 200, I would also wager that new features take a back seat and would propose that this is why we see Facebook's "upgrades" as being almost purely cosmetic (i.e. layout, markup, etc.).
Of course they have no QA... (Score:4, Interesting)
It's easy not to have QA when your users aren't paying you.
Google's in the same boat - the websites aren't the product, the end users are. When your website is your product this crap won't fly.
Ken Thompson code (Score:1, Interesting)
Ken Thompson got annoyed that Unix users would call him up or email him complaining about bugs, but would refuse to give him root access to their systems to snoop around. So he added a seemingly innocuous chunk of code to the Unix source, that would be transformed by the C compiler into a backdoor for him. Then when folks called up to complain, he would instruct them to recompile the kernel, and presto, he could get in to debug.
When describing this years later, he quipped something like, "Never trust or hire a guy like me."
So the summary says, "All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don't invite them to"
So folks can review the code . . . can they understand what it really does?
Re:no process (Score:4, Interesting)
Something TFA doesn't talk about is how the engineers are organized -- surely there's a team that does memcache, a team that does database work (like those Cassandra guys), a team that does i18n, a team that does rendering, a team that specializes in Flash doodads, etc. It's hard to see how the engineers check each other without knowing where demarcations lie.
My issue wouldn't be with the quality of the code, it's how they decide what code to write in the first place. All of the Kaizen processes work great on an assembly line, but a feature request and a coworker evaluation aren't fender panels. They require someone in authority to tell you what you're trying to accomplish in the first place. The description of the way people decide what to add (basically shutting out the marketing people if they speak out too much) and how they discipline each other (too many blames in the SVN log and too many "public shamings" result in termination) speaks to a culture where new ideas and customer focus are almost stigmatized, and where the engineers don't try to evaluate each others ideas as much as they worry about getting voted off the island.