Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we talk about below are open source so you can try them yourself.
Sapling: Scaling version control
Sapling is a version control system that can scale to huge sizes, but also emphasizes usability. There are three main components to Sapling – a server, a client, and a virtual file system.
The server stores all the data and is a careful mix of clever storage formats, wire protocols, and algorithms, mostly implemented in Rust and architected to scale.
The client then talks to that server, providing all the familiar operations (check out, rebase, commit, amend, etc). In addition, the client is also capable of talking to a git server, meaning that open source GitHub repositories can be worked on using our open-source Sapling release.
The final component is the virtual file system. When checking out a repository of our scale, just the disk I/O of writing all the files can take a significant amount of time. One solution is sparse checkouts, where a developer declares what subset of the repo they wish to see in advance. A more ergonomic alternative is EdenFS, which checks out everything in a few seconds but then only actually downloads the files from the server when they are accessed.
Buck2: Build system
After making changes, many developers at Meta use Buck2 to compile the results and test out their changes. Buck2 is designed to work at large scale, supporting remote caching and execution, so that developers can share each other’s compilations and a single developer can have access to thousands of machines to run compilations in parallel. Buck2 is also designed to support multiple programming languages simultaneously – so if you want your OCaml program to depend on a Rust library that uses a C++ library whose source code was generated by Erlang, that will work just fine.
Buck2 works well without Sapling, but has specific design considerations to enable Sapling and EdenFS. Buck2 uses Watchman to find out which files have changed, and Watchman supports EdenFS so that it can integrate smoothly with files that aren’t on the disk. Buck2 can also use specific EdenFS operations to access the file without going via the disk, optimizing performance on systems where virtual file systems can be slower.
Infer, RacerD, and Jest: Testing and static analysis
Handwritten tests and static analysis play an important role in making sure all of our code functions as intended. Working with the quantity of code we do at Meta means that we need tools that provide a high-quality signal, and do so very quickly.
For general static analysis, we use a platform called Infer, which is interprocedural and supports multiple languages, including Java and C++. We also have more tailor-made analysis tools such as RacerD, which detects Java concurrency bugs. RacerD played a big role in our project to move Facebook’s News Feed on Android from single-threaded to multi-threaded.
We also have language-specific testing frameworks. For example, Jest is our Javascript testing framework. In 2022 we officially transferred Jest to the OpenJS Foundation to further support its growth within the wider industry.
Finally, there are tools that sit between static analysis and manual test cases. Our Sapienz tool, for example, automatically tests mobile apps by allowing developers to simulate the user experience to seek out crashes and other potential issues.
Learn more about Meta’s developer workflow
In addition to our open-source tools, our developers also use a number of proprietary tools in their day-to-day workflows as well. For example, Phabricator (Phab for short), our CI and reviewing tool, helps our developers review and submit stacks of diffs. You can find more about these tools (along with the ones covered above) in the article on Meta developer’s workflow.
