Yesterday an interesting question appeared on Quora in which a user was wondering what forms of automated testing Facebook uses. Steven Grimm, Facebook’s test engineering tech lead, jumped in and provided a pretty comprehensive response. For non-programmers, be warned that his response is completely technical. However given that it was in response to a technical question, it provides superb insight for software developers.

Sten Grim gave the following response when asked about the form of automated testing completed by Facebook:

• For our PHP code, we have a suite of over 1500 test classes using the PHPUnit framework. They range in complexity from simple true unit tests to large-scale integration tests that hit our production backend services. The PHPUnit tests are run both by developers as part of their workflow and continuously by an automated test runner on dedicated hardware. Our developer tools automatically use code coverage data to run tests that cover the outstanding edits in a developer sandbox, and a report of test results is automatically included in our code review tool when a patch is submitted for review.
• For browser-based testing of our Web code, we use the Watir framework. In fact, we just hired Charley Baker, one of Watir’s lead developers, as a full-time employee. We have Watir tests covering a range of the site’s functionality, particularly focused on privacy — there are tons of “user X posts item Y and it should/shouldn’t be visible to user Z” tests at the browser level.
  In addition to the fully automated Watir tests, we have semi-automated tests that use Watir so humans can avoid the drudgery of filling out form fields and pressing buttons to get through UI flows, but can still examine what’s going on and validate that things look reasonable.
• We’re starting to use JSSpec for unit-testing JavaScript code, though that’s still in its early stages at this point.
  For backend services, we use a variety of test frameworks depending on the specifics of the services. Projects that we release as open source use open-source frameworks like Boost’s test classes or JUnit. Projects that will never be released to the outside world can use those, or can use an internally-developed C++ test framework that integrates tightly with our build system. A few projects use project-specific test harnesses. Most of the backend services are tied into a continuous integration / build system that constantly runs the test suites against the latest source code and reports the results into the results database and the notification system.
• HipHop has a similar continuous-integration system with the added twist that it not only runs its own unit tests, but also runs all the PHPUnit tests. These results are compared with the results from the same PHP code base run under the plain PHP interpreter to detect any differences in behavior.

Our test infrastructure records results in a database and sends out email notifications on failure with developer-tunable sensitivity (e.g., you can choose to not get a notification unless a test fails continuously for some amount of time, or to be notified the instant a single failure happens.) The user interface for our test result browser is integrated with our bug/task tracking system, making it really easy to associate test failures with open tasks.

A significant fraction of tests are “push-blocking” — that is, a test failure is potential grounds for holding up a release (this is at the discretion of the release engineer who is pushing the code in question out to production, but that person is fully empowered to stop the presses if need be.) Blocking a push is taken very seriously since we pride ourselves on our fast release turnaround time.

My team, Test Engineering, is responsible for building the common infrastructure used by all the above stuff, as well as for maintaining PHPUnit and Watir. Facebook has no dedicated QA team; all Facebook engineers are responsible for writing automated tests for their code and keeping the tests maintained as the underlying code changes.

I personally found the lack of a QA team to be somewhat surprising however not completely odd considering many of the automated systems in place.