Hugh Pickens writes "yeegay has a very interesting article about how Facebook develops and releases software that he has gathered from talking with friends at the company. The two largest teams at Facebook are Engineering and Ops, with roughly 400-500 team members each, together making up about 50% of the company. All engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs. After boot camp, all engineers get access to live DB and any engineer can modify any part of Facebook's code base and check-in at-will so that engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. Then arguments about whether or not a feature idea is worth doing or not generally get resolved by spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users. "All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to," writes yeegay. "It would take intentionally malicious behavior to get un-reviewed code in.” What is interesting for a compnay this size is that there is no official QA group at Facebook but almost every employee is dogfooding the product every day: many times a day and every employee is using a version of the site that includes all the changes that are next in line to go out. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon. Facebook has about 60,000 servers with the smallest level comprising only 6 servers and there are nine levels for pushing out new code. For new code the ops team observes those 6 servers at level 1 to make sure that they are behaving correctly before rolling forward to the next level. If a release is causing any issues (e.g., throwing errors, etc.) then the push is halted, the engineer who committed the offending changeset is paged to fix the problem, and then the release starts over again at level 1."