As video consumption on mobile devices grows rapidly, Facebook mobile engineers are faced with new challenges for rendering content efficiently. Videos demand more resources than simpler UI elements like text and photos. They use decoders that keep the CPU busy; they allocate large amounts of memory to set up; and they use more network bandwidth to download video data from the servers. Playing videos inside a scrollable container like News Feed is especially challenging – pressure on the device’s resources risks dropping frames, which can cause a jittery scrolling experience. In addition, we don’t want people to wait for the video to load, nor do we want it to buffer mid-playback, so the video player needs to start quickly and run smoothly. All of these challenges are intensified on Android given the wide range of devices in the market.
We recently finished migrating News Feed on Android to be powered by Litho, our open source UI rendering framework. Litho supports asynchronous layout and fine-grained recycling, which not only helped optimize News Feed to render content more efficiently, but also made our code more robust and easier to scale. We wanted to bring these benefits to video to improve the playback experience for people on Facebook and for engineers designing new use cases.
Building a video UI element was more challenging than other UI pieces we’ve worked on. The video component takes advantage of advanced features in Litho that other components don’t need, and it has inspired features that didn’t exist before. The new Litho video component introduced several improvements, from News Feed’s scroll performance to a more flexible design that can be easily reused for various video features. This post covers the challenges we faced while rewriting the video player in our Android app and how Litho helped overcome them.
Video in News Feed
News Feed supports several video story types. Some behave and look differently than others. A common story type is a Video Attachment, where regular video, Facebook Live video, 360 video, GIFs, or other formats are attached to a regular post.
Other video story types can play a generated video, a sponsored message, or a short animation.
Supporting all these variations resulted in code that was hard to maintain and test. Our main video view for video attachments was called VideoAttachmentView. For a few story types, we had to extend that view in order to reuse and customize it to fit the design. In some cases, we couldn’t make all the changes in a derived class and had to create a separate view class, which meant moving common logic to helper classes. That complicated the code further.
News Feed’s scroll performance also was impacted. RecyclerView recycles views only from the same class type, which means it can’t recycle a view like SponsoredVideoAttachmentView for a regular video attachment story, even though SponsoredVideoAttachmentView extends VideoAttachmentView. Because of this inefficiency, RecyclerView allocates more views to accommodate the different story types, contributing to a larger memory footprint. Also, the allocation is triggered right before the video view needs to appear onscreen, increasing the chances it won’t be ready in time and will drop one or more frames. Even when reusing the same VideoAttachmentView in a new layout to modify the appearance of VideoAttachmentView, we need to create a new view type. Because we’re adding another layer, this also makes the view hierarchy deeper. The deeper the view hierarchy gets, the longer it takes to render.
Designing the video component
Litho uses units called “components” to define the UI. It supports two main types of components: MountSpec and LayoutSpec. A MountSpec defines a UI building block, such as a text or image component. A LayoutSpec defines a layout containing one or more components. It’s common practice in Litho to use component composition to construct more complex components. A LayoutSpec can define a layout containing other LayoutSpec and MountSpec components, while a MountSpec renders a specific view or drawable.
A video component is a UI building block, which means we need a MountSpec to define it. One option is to create a MountSpec that renders a VideoAttachmentView, but we then would need to create a MountSpec for the other extended views, such as SponsoredVideoAttachmentView. While that could work, we’d end up with the same reuse and maintainability problems. Another approach would be to create a MountSpec that renders a simple video view instead of VideoAttachmentView, and adds it to a LayoutSpec that adjusts it for a specific story type. The following diagram illustrates the relationship between the new components:
The CoreVideoComponent is a MountSpec with the minimum features any video story would need.
@MountSpec
public class CoreVideoComponentSpec {
@OnCreateMountContent
static SimpleVideoView onCreateMountContent(ComponentContext context) {
return new SimpleVideoView(context);
}
@OnPrepare
static void onPrepare(
ComponentContext context,
@Prop VideoParams videoParams) {
prefetchVideo(videoParams);
}
@OnMount
static void onMount(
ComponentContext context,
SimpleVideoView videoView,
@Prop VideoParams videoParams) {
initVideoPlayback(videoView, videoParams);
}
@OnUnmount
static void onUnmount(
ComponentContext context,
SimpleVideoView videoView,
@Prop VideoParams videoParams) {
cleanupVideoPlayback(videoView, videoParams);
}
...
}
The CoreVideoComponent is added as a child to AutoplayVideoComponent, a LayoutSpec that registers the video to be autoplayed in News Feed. All videos in News Feed are registered to the autoplay manager, but not all videos in the app need the autoplay feature (for example, those in the full-screen video player).
@LayoutSpec
public class AutoplayVideoComponentSpec {
@OnCreateLayout
static Component onCreateLayout(
ComponentContext c,
@Prop VideoParams videoParams) {
registerVideoForAutoplay(videoParams);
return CoreVideoComponent.create(c)
.videoParams(videoParams)
.build();
}
...
}
And finally, we add the autoplay video component as a child to VideoAttachmentComponent. This component translates a video attachment data structure to a property that the more generic video component understands.
@LayoutSpec
public class VideoAttachmentComponentSpec {
@OnCreateLayout
static Component onCreateLayout(
ComponentContext c,
@Prop VideoAttachmentInfo videoAttachmentInfo) {
final VideoParams videoParams = createVideoParams(videoAttachmentInfo);
return AutoplayVideoComponent.create(c)
.videoParams(videoParams)
.build();
}
...
}
This design gives us more flexibility than VideoAttachmentView does. Any of these components can be added to another LayoutSpec, creating a more complex component that extends its functionality and/or UI design. Litho encourages the use of nested components, as well as component composition, to build larger features. The layout tree is optimized by Litho for optimal rendering performance, creating flatter views.
Here’s an example for creating a video attachment component that shows a watermark at the bottom:
@LayoutSpec
public class WatermarkVideoAttachmentComponentSpec {
@OnCreateLayout
static Component onCreateLayout(
ComponentContext c,
@Prop VideoAttachmentInfo videoAttachmentInfo) {
return Column.create(c)
.flexShrink(0)
.alignContent(YogaAlign.FLEX_START)
.child(
VideoAttachmentComponent.create(c)
.videoAttachmentInfo(videoAttachmentInfo))
.child(
Text.create(c)
.text("Powered by Litho")
.textColorRes(android.R.color.holo_green_light)
.textSizeDip(25)
.paddingDip(YogaEdge.LEFT, 5)
.positionType(YogaPositionType.ABSOLUTE)
.positionDip(YogaEdge.BOTTOM, 0)
.positionDip(YogaEdge.START, 0))
.build();
}
}
The new component reuses all the code and UI of the video attachment component by adding it as a child component.
Performance improvements
In addition to enabling a more flexible design, Litho has a few properties and features that helped us optimize the video playback in News Feed and the general performance of the entire app.
Recycling
Android’s built-in RecyclerView keeps views in different pools depending on their type, which can be problematic for UIs with many different types.
In contrast, Litho’s recycling system reuses smaller UI building blocks such as text or images rather than the entire view. By using one core video component, the same view can be recycled for all video story types. The more efficient recycling reduces object allocations and improves scroll performance.
Pre-allocation
The first video story in News Feed can’t recycle a pre-existing video view, as there were no views previously. That’s also relevant when two video stories appear onscreen: One video view can be recycled from a previous story, but the second view needs to be created at that time. When the RecyclerView needs to allocate a new view object, especially as complex as a video view, it poses a risk for dropped frames. We wanted to optimize this situation, so we created the Pre-allocation feature in Litho.
By adding a couple of properties to the MountSpec annotation, we direct Litho to allocate a few instances in advance. Pre-allocating the video views dramatically improves scroll performance when scrolling through the first video stories in News Feed.
@MountSpec(poolSize = 3, canPreallocate = true)
public class CoreVideoComponentSpec {
...
}
Life cycle
MountSpec has a few useful and simple life cycle callback methods. These let us encapsulate most of the video playback logic in the component. Before Litho, this logic was spread across different classes and was triggered by a separate controller. The main callback methods in the video component include:
- onPrepare — Starts prefetching the video. Triggered on a background thread before the video component comes onscreen.
- onMount — Initializes the video player. Triggered the first time the component is configuring its view.
- onUnmount — Cleans up the video player for its next use. Triggered when the video is scrolled away.
LayoutSpec has one main callback: onCreateLayout(). Its main purpose is to construct the layout of the LayoutSpec, but it can also prepare resources for its child components. For example, a Cover Photo LayoutSpec can create a layout with a video and a cover photo above it, while also triggering the cover photo’s prefetching, all under the same callback method.
MountSpec supports another useful callback: shouldUpdate(). When the RecyclerView’s adapter is updated, it could rebind all its child views and get all the visible components to remount (triggering onUnmount , then onMount). For simpler components, there’s no noticeable impact, but reconfiguring a video player is a heavier operation. This callback is called before Litho remounts the component, giving us a chance to skip it if it’s unnecessary (for instance, when mounting the same video).
@ShouldUpdate(onMount = true)
static boolean shouldUpdate(@Prop Diff<VideoParams> videoParamsDiff) {
return videoParamsDiff.getNext().videoId != videoParamsDiff.getPrevious().videoId;
}
Testability
The modularity of the new components helps test video playback logic more easily. Litho comes with a testing framework that simulates a component’s life cycle in a unit test. By encapsulating more of the video playback logic within these components, we can test and verify complex scenarios that we couldn’t before. In addition, extending functionality through the use of composition, instead of inheritance, is safer and easier to maintain.
Conclusion
Litho helped improve the performance, efficiency, extensibility, and maintainability of our video implementation over one built with traditional Android views. The video component has improved our scrolling performance by up to 20 percent across the Facebook for Android app. It also improved our cold start time and reduced our total out-of-memory crashes by 2.5 percent, both a result of more efficient memory management. The improvements in the code not only improve the Facebook experience, but also give engineers more leeway to create new video experiences across the Facebook app.
Litho allows us to write highly optimized UI in Android. It was open-sourced in April 2017 and hasn’t stopped growing since. At the end of last year, we launched Sections, an API for writing highly optimized list surfaces, built on top of Litho. We saw dramatic performance improvements for some of our conversions to Litho and Sections, such as scroll performance improvement as high as 42 percent. Getting started with Litho is simple and well documented. Check out the Litho website and GitHub page for code samples, a tutorial, and advanced guides.
