Birth of “OpenBMC”
At Facebook, reliability and rapid iteration are two important tenets of our work. This makes feature velocity and the ability to troubleshoot our own systems significant items as we build out our infrastructure. Recently, we found the Baseboard Management Controller (BMC) software stack too closed to meet these needs, so we built our own version, which we’re open-sourcing today. How we arrived at this open software solution is a good study in Facebook’s hacker culture — and here’s the backstory on how we got here.
For context, a BMC is a specialized controller embedded in servers. It often comes in the form of a system-on-chip (SoC), with its own CPU, memory, and storage and lots of IO. A BMC connects to sensors to read environmental conditions and to fans to control temperature. It also provides other system management functions, including remote power control, serial over LAN, and monitoring and error logging of the server host CPU and memory.
Until now, on the software side, the complete BMC software stack was closed. BMC software is usually developed by the hardware manufacturer during the hardware-development phase. Because the BMC software was closed, whatever was developed for the existing hardware could not be reused for the next generation. The long BMC software schedule directly affected new hardware development. When hardware development ended, the BMC software development stopped as well. Further bug fixes or new features had to wait for the hardware manufacturer.
When we were developing Facebook’s top-of-rack “Wedge” switch, we followed our usual process in the beginning; our partner was responsible for developing the BMC software. However, in the first months of the project, many requirements for the BMC software emerged, introducing extra complexity, coordination, and delays into the BMC software-development process.
To address these challenges, at one of Facebook’s hackathon events, four engineers worked to create our own BMC software. Within 24 hours, we were able to build a minimum BMC software image, including an SSH server and the ability to change fan speed, power-on the host CPU, and blink some LEDs. It was far from a production image, but it gave us a strong confidence that we could eventually develop our own BMC software for “Wedge.”
Fast-forward eight months, and we’ve deployed our solution — code-named “OpenBMC” — into production along with Wedge. And today we’re sharing OpenBMC with the open source community in the hope that we can collaborate based on this open software framework for next-generation system management.
“OpenBMC” under the hood
By itself, BMC hardware is a computer system. Compared with modern computer systems, the hardware resources in a BMC are very limited. A slow CPU, less than 32 MB flash as storage, and less than 256 MB RAM are common in BMC hardware. Because of this, OpenBMC is designed as a complete Linux distribution, with flexibility to be customized to support different BMC SoC or boards.
The OpenBMC image includes a bootloader (u-boot), a Linux kernel, open source packages, and board-specific packages:
- Both the bootloader and the Linux kernel include various hardware drivers for the BMC SoC, including an i2c driver, USB driver, PWM driver, and SPI driver.
- The open source packages include common applications such as BusyBox, i2c-tools, lm-sensors, OpenSSH, and Python.
- The board-specific packages include initialization scripts and tools for a particular board. For example, it includes a tool to dump asset info from the EEPROM and a fan-controller daemon to control the fan speed based on environment readings.
The following diagram illustrates “OpenBMC” packages running on the BMC inside “Wedge.”
All packages in OpenBMC are built from source by using the Yocto Project. In Yocto, software packages are grouped together into recipes, and different recipes are then grouped together into layers.
In OpenBMC, there are three sets of layers:
- The common layer includes packages that can be used in different boards and BMC SoCs.
- The SoC layer includes packages specific to BMC SoCs, such as the contribution from Aspeed for AST2300/AST2400 SoC, together with the enhancements from Facebook. Both the bootloader and the kernel are defined in the SoC layer.
- The board layer includes packages for different boards. Specifically, we are releasing the hardware configuration, initialization scripts, and tools specific to Wedge.
Building a customized OpenBMC image for a specific board involves combining the common layer, an SoC layer for the BMC SoC used in the board, and a board layer specific to the board.
We have released the OpenBMC repository through GitHub.
Next Phase
OpenBMC is in an early phase, but we believe in building a strong OpenBMC ecosystem, and so we have gone ahead and shared this early working code.
We recently restructured the code to better support different SoCs and boards. There are some applications developed for Wedge that can be reused for other boards. In order to do so, we will need to add a hardware abstraction between the applications and the hardware.
With Wedge being proposed as a contribution to the Open Compute Project, application developers can use open hardware as the development platform for OpenBMC. We can envision contributions at all the different layers of OpenBMC:
- Innovation in system management application can be contributed to the common layer. For example, a new application that could change the direction of system management could be contributed to the common layer, which would then benefit all boards powered by OpenBMC.
- BMC SoC vendors could contribute their SoC support through the SoC layer, which helps the adoption of the BMC SoC.
- Through the board layer, board manufacturers could work with the community to continuously improve the BMC software running on their boards.
Wedge is the first hardware powered by OpenBMC, and 6-pack will be the second. We are excited to share OpenBMC with the community and work toward next-generation system management.
Many people contributed to the work on this project. Credit should go to: Kevin Lahey, Sai Dasari, Aaron Miller, Adam Simpkins, and Tian Fang.
