Automatically push commits to GitHub with FBShipIt

We're excited to announce that today we're open-sourcing FBShipIt, a tool we developed internally to help manage our projects on GitHub. FBShipIt automates pushing commits to GitHub, giving our projects a consistent approach, and reducing the time engineers need to spend on this repetitive task. It also makes continuous pushing a possibility, reducing the difficulty of maintaining interdependent projects.

For most projects, the process for releasing code is lengthy and manual:

Move files: At Facebook, some of our projects are in nested subdirectories of shared repositories, but on GitHub they should be top-level.
Remove some files: These can be confidential, not useful externally, or simply part of another project.
Remove anything confidential from commit messages: “Test Plan: did x, y, z with SuperAwesomeProject.”
Replace @internal_user_name with @github_user_name in commit messages.
Manage incompatibilities between Git and Mercurial because many of Facebook's projects are now in now in Mercurial repositories.

Simply put, FBShipIt takes these steps and automates them, handling thousands of pushes a day.

Life before FBShipIt

FBShipIt started out in 2013 as “fboss” (FB Open Source Sync, unrelated to our current FB Open Switching System project) and solved this manual-commit problem just for HHVM: It copied commits between two local checkouts.

As HHVM evolved, fboss needed to evolve as well: HHVM has several rapidly developed dependencies (Folly in particular), and we needed to get changes to GitHub at roughly the same time. Coordinating releases was painful, so we decided for something completely different: continuous, unsupervised pushes. And so “fboss2” was born, introducing idempotency and repository management, and designed to support multiple projects.

This move gave us several benefits:

Engineers have more time to improve their projects.
Related and interdependent changes are now available nearly simultaneously.
Pull requests are more likely to apply cleanly, as GitHub master is up-to-date.

Until this point, changes to Folly would often require coordination between the Folly, HHVM, Thrift, and Proxygen teams to update master to compatible points. In parallel, we started work on oss_sync_and_push, a wrapper for fboss2 that added support for building and testing before pushing, so that we'd stop pushing updates that would break the build for external users.

Expansion to a general-purpose tool

At the start of 2015, most Facebook projects that synced to GitHub either had their own scripts or a developer had to do it by hand.

Throughout the year, Facebook's usage of Mercurial rapidly expanded, which meant most of these had to be rewritten to handle reading Mercurial diffs but writing Git diffs. Before this change, we were copying changes between two git repositories; with Mercurial, we needed to translate Mercurial patches to a format that Git understands. While these are mostly the same (especially with hg --git), there are several small differences, including how renamed files are represented.

As fboss2 already supported multiple projects and cleanly dealt with the Mercurial differences, we decided to migrate more projects to it instead. As this migration progressed, we also merged oss_sync_and_push, creating FBShipIt.

FBShipIt is currently used by FBShipIt itself, React Native, HHVM, Infer, Nuclide, and over 20 others.

Buck and Nuclide also use FBShipIt to automatically update their gh-pages branches; in total, FBShipIt is now automatically handling thousands of pushes a day.

How it works

FBShipIt is a Hack library that divides the process into several phases:

Clone the repositories, if necessary.
Fetch any updates.
Copy and filter commits.
Push changes to remote repository.
Any additional project-specific phases, such as building and testing.

Filtering commits is based on functions that take a Changeset object and return a new, modified one. This makes issues easy to debug.

The project-specific code is short, clear, and testable:


<?hh
/**
 * Copyright (c) 2016-present, Facebook, Inc.
 * All rights reserved.
 *
 * This source code is licensed under the BSD-style license found in the
 * LICENSE file in the root directory of this source tree. An additional grant
 * of patent rights can be found in the PATENTS file in the same directory.
 */
namespace Facebook\ShipIt;

require_once(__DIR__.'/../autoload.php');

final class ShipItFBCuda extends FBShipItCLI {
  const string ROOT = 'fbcode/cuda/';

  <<__Override>>
  protected static function projectFilterChangeset(
    ShipItChangeset $changeset,
  ): ShipItChangeset {
    return $changeset
      |> FBCommonFilters::applyAll($$)
      |> ShipItPathFilters::moveDirectories($$, ImmMap { self::ROOT => '' });
  }

  <<__Override>>
  public static function getStaticConfig(): FBShipItCLIStaticConfig {
    return shape(
      'internalRepo' => 'fbsource',
      'githubOrg' => 'facebook',
      'githubProject' => 'fbcuda',
      'sourceRoots' => ImmSet { self::ROOT },
    );
  }
}

// Allow require() from unit test
if (isset($argv) && realpath($argv[0]) === realpath(__FILE__)) {
  ShipItFBCuda::cliMain();

Additional code is required per organization, unless you are syncing between two GitHub repositories/branches. For reference, we have included Facebook's library code and the project-specific unit tests.