Friend Bubbles: Enhancing Social Discovery on Facebook Reels
Friend bubbles in Facebook Reels highlight Reels your friends have liked or reacted to, helping you discover new content and making it easier to connect over shared interests. This article explains the technical architecture behind friend bubbles, including how machine learning estimates relationship strength and ranks content your friends have interacted with to create more [...] Read More... The post Friend Bubbles: Enhancing Social Discovery on Facebook Reels appeared first on Engineering at Meta.
Friend bubbles in Facebook Reels highlight Reels your friends have liked or reacted to, helping you discover new content and making it easier to connect over shared interests. This article explains the technical architecture behind friend bubbles, including how machine learning estimates relationship strength and ranks content your friends have interacted with to create more opportunities for meaningful engagement and connection. Friend bubbles enhance the social experience on Facebook Reels by helping you discover content your friends enjoy, creating a shared viewing experience and sparking new conversations. With a quick tap on a bubble, you can start a one-on-one conversation with any friend who has engaged with that Reel. This feature combines social and interest signals to recommend more relevant, personalized content while making it easier to start conversations with the people who matter most to you. When videos connect to both personal interests and friend-related interests, they create a feedback loop that improves recommendations and strengthens social connections. An Overview of the Friend Bubbles System Architecture The friend bubbles recommendation system includes several components that work together to surface relevant, friend-interacted content by blending video-quality signals with social-graph signals: Viewer-Friend Closeness (Whose Interactions Matter Most): Identifies which friends’ interactions are most likely to interest the viewer. Video Relevance (What Videos to Show): Ranks videos that are contextually relevant to the viewer. Multiple friend interactions on the same video often signal stronger shared interest and higher relevance. Content surfaced through friend connections also tends to be high quality, creating a reinforcing loop: Social discovery increases engagement, and that engagement further strengthens the social graph. <img decoding="async" class="alignnone size-full wp-image-23762" src="https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png" alt="" width="1920" height="1203" srcset="https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png 1920w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=916,574 916w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=768,481 768w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=1024,642 1024w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=1536,962 1536w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=96,60 96w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-1.png?resize=192,120 192w" sizes="(max-width: 992px) 100vw, 62vw" /> Viewer-Friend Closeness: Identifying Friends With User-User Closeness Models Friend bubbles rely on two complementary machine learning models to identify which connections a person feels closest to. One model is based on user survey feedback; the other is based on on-platform interactions. The survey-based closeness model draws on a broad set of signals, including social-graph features (mutual friends, connection strength, interaction patterns) and user attributes (behavioral and demographic signals such as user-provided location, number of friends, and number of posts shared) to build a more complete picture of real-world relationships. It is trained on a regular cadence using a lightweight binary survey in which a randomly selected group of Facebook users is asked whether they feel close to a specific connection in real life. The survey is structured as a close vs. not-close prediction problem, refreshed regularly to keep labels current, and includes questions that act as proxies for offline relationship strength (such as how often two people communicate). In production, the model runs weekly inference over trillions of person-to-person connections across Facebook friends. While survey-based closeness provides a strong foundation, friend bubbles also use a context-specific closeness prediction model trained on on-platform activity signals, using real interactions that occur when bubbles are shown (for example, likes, comments and reshares). This enables the model to capture closeness in context — how likely a viewer is to value content recommended by someone in their friend graph based on how they interact with each other on the platform. Our approach emphasizes connection quality over quantity. While bubble prevalence naturally rises with larger friend graphs, showing more bubble videos does not necessarily increase user engagement. The goal is to surface the right friend connections — those most likely to make the social context meaningful — using a combination of existing closeness signals and surface-specific features that better reflect the relationship dynamics behind friend-driven recommendations. Video Relevance: Making the Ranking System Friend-Content Aware We use two key strategies to ensure high-quality, friend-interacted content can move through the recommendation funnel and reach users: expanding the top of the funnel, and enabling models to rank friend-bubble content effectively through a continuous feedback loop. <img decoding="async" class="alignnone size-full wp-image-23763" src="https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png" alt="" width="1999" height="1125" srcset="https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png 1999w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=916,516 916w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2026/03/Facebook-Friend-Bubbles_image-3.png?resize=192,108 192w" sizes="(max-width: 992px) 100vw, 62vw" /> Sourcing Inventory: Expanding the Top of Funnel The retrieval stage sources candidate videos based on close friends, as identified by the closeness model described above. By explicitly retrieving friend-interacted content, we expand the top of the funnel to ensure sufficient candidate volume for downstream ranking stages. This is important because, without it, high-quality friend content may never enter the ranking pipeline in the first place. Enabling Models to Rank Friend Content Effectively Through a Continuous Feedback Loop A key insight from our development process was understanding why friend-interacted videos sometimes struggled to rank highly: It wasn’t because they were low quality, but because the model lacked user-user closeness context. Without that context, the model can’t learn what makes friend content uniquely valuable — namely, that its relevance is often driven by relationship strength and social meaning rather than the same signals that explain interest in more general content. To address this gap, we integrated friend-bubble interaction signals as features and added new tasks into both early-stage and late-stage ranking multi-task, multi-label (MTML) models to incorporate viewer-friend relationship strength and to learn downstream engagement on videos with social bubbles. With these signals added across the ranking funnel, the models can better recognize the value of friend-interacted content, learn the relationship between closeness and viewer interest, and rank high-quality friend content higher when it is most relevant. The system includes a continuous feedback loop in which friend-bubble interaction data flows back into model training. This loop helps the ranking system improve its understanding of which friend-content combinations resonate with users. We augmented our existing video-ranking formula, which includes several optimization goals, with a friend-bubble ranking objective designed to maximize overall video engagement. We consider interaction metrics such as watch time, comments and likes, and use a conditional probability term, P(video engagement | bubble impression), to predict the likelihood that a user will engage with a video after seeing a friend bubble. This is balanced with tunable weights that manage trade-offs between social interaction and video engagement, allowing us to optimize for social connection (helping people discover videos their friends like) and content quality. This dual optimization captures the core value proposition of the friend-content ranking system: enabling effortless connection through passive friend discovery, delivering entertainment through relevant content, and strengthening relationships by turning shared videos into natural touchpoints for conversation. Client Infra Behind the Scenes: Performance at Reels Scale Reels is a performance-sensitive surface, so adding new per-video metadata isn’t as simple as adding another field. If it increases work during scrolling or delays playback, it can hurt the core user experience. When we integrated friend bubbles, we treated three constraints as nonnegotiable: Smooth scrolling No regressions in load latency Low CPU overhead for metadata fetch and processing Facebook’s video delivery system already performs significant prefetch work ahead of playback. It preloads metadata, thumbnails and buffered content before a video reaches the viewport. We pinned friend-bubble metadata retrieval to that same prefetch window, which gave us several benefits: We could reuse cached results for stable data, avoid redundant CPU work, and limit wasted network requests by using an already optimized fetch path. Because the bubble data arrived alongside the video content, we could render bubbles at the same time as the video itself, eliminating mid-playback UI updates and redraws. We also made animation strictly conditional. During active scrolling and interaction, animation is disabled to preserve scroll responsiveness. On low-end devices where even idle animation could compromise performance, we turn it off entirely. Along with additional optimizations in the underlying method, this approach enabled us to ship friend bubbles while preserving core Reels performance. Why the Metadata Has to Earn Its Place A cleaner user interface is usually better, and new metadata can backfire if it adds noise or slows the experience. Friend bubbles work because the signal is high value: It adds meaningful social context that helps people decide what’s worth watching. By setting a conservative threshold for which friends are eligible to appear, we ensure bubbles show up only when the relationship signal, as determined by the user-user closeness model, is strong. That approach reduces clutter while improving the viewing experience overall, reflected in increased video watch time. The Impact and Future of Friend Bubbles Friend bubbles improve content relevance and engagement quality. In user feedback surveys, bubble-annotated videos consistently receive higher interest scores and more positive sentiment ratings than videos without bubbles. Beyond relevance, bubbles improve app-session quality, not just quantity. Users who see bubbles spend more time actively watching and interacting with content, with growth concentrated in longer sessions rather than brief check-ins. The improvements come primarily from deeper video consumption. Bubble-related signals show a delayed effect on longer-term engagement patterns, suggesting repeated exposure to content friends have interacted with builds sustained interest over time. By surfacing content friends have engaged with, bubbles also expose users to a broader range of topics and creators than they would otherwise encounter organically. Users don’t just passively scroll past this content — they actively engage through likes, comments, shares and follows, indicating friend-recommended content can resonate even when it falls outside their typical interests. Not all friend signals are equal. Bubbles triggered by expressive reactions such as love or laughter drive stronger downstream engagement than simple likes, particularly for comments and private shares, suggesting expressive reactions signal stronger resonance. Engagement also scales consistently with the number of friend bubbles shown, meaning videos with multiple friend interactions tend to perform better. Next, we’re scaling the system to increase impact and robustness by expanding friend-driven recommendations — while preserving quality — to additional surfaces and inventory, improving cold start for people with limited friend graphs, and refining ranking and feedback signals for better personalization. Ultimately, this architecture illustrates how machine learning can strengthen human connection at scale, helping people discover shared interests and making it easier to start conversations with the people who matter most. When your friends enjoy something great, you can discover it, too — and you’re only a tap away from talking about it together. For more information about Facebook Bubbles, visit the Meta Newsroom. The post Friend Bubbles: Enhancing Social Discovery on Facebook Reels appeared first on Engineering at Meta.
Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation
Meta’s Ranking Engineer Agent (REA) autonomously executes key steps across the end-to-end machine learning (ML) lifecycle for ads ranking models. This post covers REA’s ML experimentation capabilities: autonomously generating hypotheses, launching training jobs, debugging failures, and iterating on results. Future posts will cover additional REA capabilities. REA reduces the need for manual intervention. It manages [...] Read More... The post Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation appeared first on Engineering at Meta.
Meta’s Ranking Engineer Agent (REA) autonomously executes key steps across the end-to-end machine learning (ML) lifecycle for ads ranking models. This post covers REA’s ML experimentation capabilities: autonomously generating hypotheses, launching training jobs, debugging failures, and iterating on results. Future posts will cover additional REA capabilities. REA reduces the need for manual intervention. It manages asynchronous workflows spanning days to weeks through a hibernate-and-wake mechanism, with human oversight at key strategic decision points. In its first production rollout, REA delivered: 2x Model Accuracy: REA-driven iterations doubled average model accuracy over baseline across six models. 5x Engineering Output: With REA-driven iteration, three engineers delivered proposals to launch improvements for eight models — work that historically required two engineers per model. The Bottleneck in Traditional ML Experimentation Meta’s advertising system delivers personalized experiences to billions of people across Facebook, Instagram, Messenger, and WhatsApp. Powering these interactions are highly sophisticated, complex and massively distributed machine learning (ML) models that continuously evolve to serve both advertisers and people who use the platforms. Optimizing these ML models has traditionally been time-consuming. Engineers craft hypotheses, design experiments, launch training runs, debug failures across complex codebases, analyze results and iterate. Each full cycle can span days to weeks. As Meta’s models have matured over the years, finding meaningful improvements has become increasingly challenging. The manual, sequential nature of traditional ML experimentation has become a bottleneck to innovation. To address this, Meta built the Ranking Engineer Agent, an autonomous AI agent designed to drive the end-to-end ML lifecycle and iteratively evolve Meta’s ads ranking models at scale. Introducing REA: A New Kind of Autonomous Agent Many AI tools used in ML workflows today function as assistants: They are reactive, task-scoped and session-bound. They can help with individual steps (for example, drafting a hypothesis, writing configuration files, interpreting logs), but they typically cannot run an experiment end to end. An engineer still has to decide what to do next, re-establish context, and drive progress across long-running jobs — and debug inevitable failures. REA is different: an autonomous agent built to drive the end-to-end ML lifecycle, coordinating and advancing ML experiments across multiday workflows with minimal human intervention. REA addresses three core challenges in autonomous ML experimentation: Long-Horizon, Asynchronous Workflow Autonomy: ML training jobs run for hours or days, far beyond what any session-bound assistant can manage. REA maintains persistent state and memory across multiround workflows spanning days or weeks, staying coordinated without continuous human supervision. High-Quality, Diverse Hypothesis Generation: Experiment quality is only as good as the hypothesis that drives it. REA synthesizes outcomes from historical experiments and frontier ML research to surface configurations unlikely to emerge from any single approach, and it improves with every iteration. Resilient Operation Within Real-World Constraints: Infrastructure failures, unexpected errors and compute budgets can’t halt an autonomous agent. REA adapts within predefined guardrails, keeping workflows moving without escalating routine failures to humans. REA addresses these challenges through a Hibernate-and-Wake Mechanism for continuous multiweek operation, a Dual-Source Hypothesis Engine that combines a historical insights database with a deep ML research agent, and a Three-Phase Planning Framework (Validation → Combination → Exploitation) that operates within engineer-approved compute budgets. How REA Manages Multi-Day ML Workflows Autonomously REA is built around a core insight: Complex ML optimization isn’t a single task. It is a multistage process that unfolds over days or weeks. The agent must reason, plan, adapt and persist across this entire horizon. Long-Horizon Workflow Autonomy Traditional AI assistants operate in short bursts, responding to prompts and then waiting for the next query. ML experimentation doesn’t work that way. Training jobs run for hours or days, and the agent must remain coordinated across these extended timelines. REA uses a hibernate-and-wake mechanism. When the agent launches a training job, it delegates the wait to a background system, shuts down to conserve resources, and automatically resumes where it left off when the job completes. This enables efficient, continuous operation across extended time frames without requiring constant human monitoring. To support this, Meta built REA on an internal AI agent framework, Confucius, designed for complex, multistep reasoning tasks. It provides strong code generation capabilities and a flexible SDK for integrating with Meta’s internal tooling systems, including job schedulers, experiment tracking infrastructure and codebase navigation tools. High-Quality, Diverse Hypothesis Generation The quality of the hypothesis directly determines the quality of an ML experiment. REA consults two specialized systems to generate diverse, high-quality ideas: Historical Insights Database: A curated repository of past experiments that enables in-context learning and pattern recognition across prior successes and failures. ML Research Agent: A deep research component that investigates baseline model configurations and proposes novel optimization strategies, using Meta’s historical insights database. By synthesizing insights from both sources, REA surfaces configurations unlikely to emerge from any single approach in isolation. REA’s most impactful improvements have combined architectural optimizations with training-efficiency techniques — a result of this cross-system methodology. Resilient Execution Within Real-World Constraints Real-world experimentation operates under compute constraints and inevitable failures. REA addresses both through structured planning and autonomous adaptation. Before executing any plan, REA proposes a detailed exploration strategy, estimates total GPU compute cost, and confirms the approach with an engineer. A typical multiphase plan proceeds through three stages: Validation: Individual hypotheses from different sources are tested in parallel to establish quality baselines. Combination: Promising hypotheses are combined to search for synergistic improvements. Exploitation (Intensive Optimization): The most promising candidates are explored aggressively to maximize results within the approved compute budget. When REA encounters failures — such as infrastructure issues, unexpected errors, or suboptimal results — it adjusts the plan within predefined guardrails instead of waiting for human intervention. It consults a runbook of common failure patterns, makes prioritization decisions (such as excluding jobs with clear out-of-memory errors or training instability signals such as loss explosions), and debugs preliminary infrastructure failures from first principles. This resilience is critical for maintaining autonomy over long-horizon tasks, where engineers provide periodic oversight rather than continuous monitoring. REA operates with rigorous safeguards. It works exclusively on Meta’s ads ranking model codebase. Engineers grant explicit access controls through preflight checklist reviews, and REA confirms compute budgets up front, halting or pausing runs when thresholds are reached. The REA System Architecture <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23734" src="https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg" alt="" width="1463" height="720" srcset="https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg 1463w, https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg?resize=916,451 916w, https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg?resize=768,378 768w, https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg?resize=1024,504 1024w, https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg?resize=96,47 96w, https://engineering.fb.com/wp-content/uploads/2026/03/Meta-REA_image-2.jpg?resize=192,94 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> The Ranking Engineer Agent is built on two interconnected components, REA Planner and REA Executor, supported by a shared Skill, Knowledge and Tool System that provides ML capabilities, historical experiment data, and integrations with Meta’s internal infrastructure. Together, they directly enable the agent’s three core capabilities. Long-Horizon Autonomy is powered by the execution flow: An engineer collaborates with the hypothesis generator to create a detailed experiment plan through the REA Planner. That plan is exported to the REA Executor, which manages asynchronous job execution through an agent loop and wait state, entering a wait state during training runs and resuming with results upon completion rather than requiring continuous human monitoring across multiweek workflows. High-Quality, Diverse Hypothesis Generation is driven by the knowledge flow: As the executor completes experiments, a dedicated experiment logger records outcomes, key metrics, and configurations into a centralized hypothesis experiment insight database. This persistent memory accumulates knowledge across the full history of the agent’s operation. The hypothesis generator draws on these insights to identify patterns, learn from prior successes and failures, and propose increasingly sophisticated hypotheses for each subsequent round, closing the loop and compounding the system’s intelligence over time. Resilient Execution is maintained across both flows: When the executor encounters failures, infrastructure errors, out-of-memory signals, or training instability, it consults a runbook of common failure patterns and applies prioritization logic to adapt autonomously within predefined guardrails. It then resumes the planner with actionable results rather than surfacing routine interruptions to engineers. Impact: Model Accuracy and Engineering Productivity 2x Model Accuracy Over Baseline Approaches In the first production validation across a set of six models, REA-driven iterations doubled average model accuracy over baseline approaches. This translates directly to stronger advertiser outcomes and better experiences on Meta platforms. 5x Engineering Productivity Gains REA amplifies impact by automating the mechanics of ML experimentation, enabling engineers to focus on creative problem-solving and strategic thinking. Complex architectural improvements that previously required multiple engineers over several weeks can now be completed by smaller teams in days. Early adopters using REA increased their model-improvement proposals from one to five in the same time frame. Work that once took two engineers per model now takes three engineers across eight models. The Future of Human-AI Collaboration in ML Engineering REA represents a shift in how Meta approaches ML engineering. By building agents that can autonomously manage the entire experimentation lifecycle, the team is changing the structure of ML development — moving engineers from hands-on experiment execution toward strategic oversight, hypothesis direction, and architectural decision-making. This new paradigm, where agents handle iterative mechanics while humans make strategic decisions and final approvals, is just the beginning. Privacy, security, and governance remain key priorities for the agent. Meta continues to enhance REA’s capabilities by fine-tuning specialized models for hypothesis generation, expanding analysis tools, and extending the approach to new domains. Acknowledgements Ashwin Kumar, Harpal Bassali, Shashank Ankit, Deepak Chandra, Chaorong Chen, Wenlin Chen, Vitor Cid, Peter Chu, Xiaoyu Deng, Jingyi Guan, Junhua Gu, Liquan Huang, Qinjin Jia, Santanu Kolay, Jakob Moberg, Shweta Memane, Jp Owed, Sandeep Pandey, Vijay Pappu, Shyam Rajaram, Ben Schulte, Jags Somadder, Matt Steiner, Ritwik Tewari, Hangjun Xu, Zhaodong Wang, Fan Yang, Xin Zhao, Zoe Zu The post Ranking Engineer Agent (REA): The Autonomous AI Agent Accelerating Meta’s Ads Ranking Innovation appeared first on Engineering at Meta.
Patch Me If You Can: AI Codemods for Secure-by-Default Android Apps
Even seemingly simple engineering tasks — like updating an API — can become monumental undertakings when you’re dealing with millions of lines of code and thousands of engineers, especially if the changes are security-related. Nowhere is this more apparent than in mobile security, where a single class of vulnerability can be replicated across hundreds of [...] Read More... The post Patch Me If You Can: AI Codemods for Secure-by-Default Android Apps appeared first on Engineering at Meta.
Even seemingly simple engineering tasks — like updating an API — can become monumental undertakings when you’re dealing with millions of lines of code and thousands of engineers, especially if the changes are security-related. Nowhere is this more apparent than in mobile security, where a single class of vulnerability can be replicated across hundreds of call sites scattered throughout a sprawling, multi-app codebase serving billions of users. Meta’s Product Security team has developed a two-pronged strategy to address this: Designing secure-by-default frameworks that wrap potentially unsafe Android OS APIs and make the secure path the easiest path for developers, and Leveraging generative AI to automate the migration of existing code to those frameworks at scale. The result is a system that can propose, validate, and submit security patches across millions of lines of code with minimal friction for the engineers who own them. On this episode of the Meta Tech Podcast, Pascal Hartig talks to Alex and Tanu, from Meta’s Product Security team about the challenges and learnings from the journey of making Meta’s mobile frameworks more secure at a scale few companies ever experience. Tune in to this episode and join us as we explore the compelling crossroads of security, automation, and AI within mobile development. Download or listen to the episode below: You can also find the episode wherever you get your podcasts, including: Spotify Apple Podcasts Pocket Casts The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features. Send us feedback on Instagram, Threads, or X. And if you’re interested in learning more about career opportunities at Meta visit the Meta Careers page. The post Patch Me If You Can: AI Codemods for Secure-by-Default Android Apps appeared first on Engineering at Meta.
How Advanced Browsing Protection Works in Messenger
We’re sharing the technical details behind how Advanced Browsing Protection (ABP) in Messenger protects the privacy of the links clicked on within chats while still warning people about malicious links. We hope that this post has helped to illuminate some of the engineering challenges and infrastructure components involved for providing this feature for our users. [...] Read More... The post How Advanced Browsing Protection Works in Messenger appeared first on Engineering at Meta.
We’re sharing the technical details behind how Advanced Browsing Protection (ABP) in Messenger protects the privacy of the links clicked on within chats while still warning people about malicious links. We hope that this post has helped to illuminate some of the engineering challenges and infrastructure components involved for providing this feature for our users. While end-to-end encryption (E2EE) on Messenger ensures that direct messages and calls are protected, Messenger’s Safe Browsing feature safeguards against malicious links within end-to-end encrypted messages and calls on the app. If you’re sent an unsafe link for some reason – maybe it’s sent by someone you don’t know or by a friend whose account has been compromised – Safe Browsing warns you that the link points to an unsafe website that may try to steal passwords or other personal information from you. In its standard setting, Safe Browsing uses on-device models to analyze malicious links shared in chats. But we’ve extended this further with an advanced setting called Advanced Browsing Protection (ABP) that leverages a continually updated watchlist of millions more potentially malicious websites. To build ABP, we had to leverage a series of intricate infrastructure components, a complex system of cryptographic primitives, all working together with the goal of protecting user privacy in Messenger. Private Information Retrieval – The Starting Point for ABP ABP closely mirrors the setting for a cryptographic primitive known as private information retrieval (PIR). In the classical PIR setting, a client queries a server (that holds a database) to learn whether or not the subject of the query is a member of that database. This protocol aims for the server to learn as little information as possible (ideally no information) about the client’s query. In a theoretical setting, the server could send the entire database to the client, allowing the client to perform subsequent query lookups on its own, without needing to involve the server anymore. However, the database used by ABP needs to be updated frequently, and is too large to reasonably be sent down to the client. Furthermore, revealing the entire database to the client could inadvertently aid attackers attempting to circumvent the system. Other work has suggested that this approach can be improved upon by using an oblivious pseudorandom function (OPRF) and dividing the database into multiple shards (or “buckets”) so that the linear-time operation is performed over a fraction of the database. This existing approach was the starting point for our implementation of ABP, but there were two issues that we needed to adapt into our setting. An OPRF works well for queries that are exact matches into the database. However, URL-matching queries are not exact matches, as we will describe in more detail, shortly. This also means that the client still needs to tell the server which bucket to look into. This inherently introduces a tradeoff between the privacy of the system versus the efficiency/bandwidth: The less granular the buckets, the less efficient the protocol becomes, but the less information is leaked from client’s query to the server. There are also other approaches, namely cryptographic constructions, which improve this tradeoff by employing lattice-based techniques to reduce the amount of sharding needed. However, at the time of writing, these did not appear to be practical enough to completely eliminate the need for sharding at our scale. This could be a promising future direction for the system, though, and for industrial applications of PIR in general. How ABP Handles Prefix Queries for URLs The server’s database entries consist of URL domains with (and without) paths, which do not always correspond to exact link matches. For instance, if an entry for “example.com” existed in our database, and the client submits a query in the form, “example.com/a/b/index.html” this should be reported to the client as a match, even though the link contents do not match exactly. Instead, what we need is a privacy-preserving “URL-matching” scheme between the client’s query and each of the database entries. Subdomains are also a consideration here, but we’ve omitted them for the simplicity of this example. One simple approach we considered to address these prefix queries was to run a series of parallel queries for PIR, one for each path prefix of the URL. So, in our running example of the client query being “example.com/a/b/index.html” the client would create PIR queries for: example.com example.com/a example.com/a/b example.com/a/b/index.html Functionally, this would satisfy prefix matching, but there is a privacy issue with this approach: Each of these path prefix queries leaks extra information about the client’s actual URL. If the PIR scheme we use does not leak any information to the server, then this might be acceptable, but if the server learns B bits of the client query, then in this scheme the server learns P * B bits, where P is the number of path prefixes in the URL. For extremely long URLs, this might even be enough to uniquely identify a plaintext link! In order to reduce the leakage to the server, we can instead have the server group together links that share the same domain. This way, the client can again request just one bucket (the bucket corresponding to the URL’s domain), then check all the prefix URL path components for membership in that one bucket. This would indeed address the privacy issue so that the server only learns B bits. But it also creates a new efficiency problem: Bucket sizes can become unbalanced. We create buckets by hashing URLs. If we were to hash full URLs, we could expect bucket sizes to be approximately uniform because each blocklist entry is mapped to a bucket pseudorandomly. When we hash only domains, that’s no longer the case. If many blocklist entries share the same domain, they all end up in the same bucket. It turns out that in practice many blocklisted URLs do share domains. For example, consider link shortening services: These services might host many, many URLs (both malicious and benign) that all share the same domain. If many links share the same domain and, hence, belong in the same bucket, then the size of the bucket might be too large to be able to return to the client. And since we apply padding to buckets, the response size would be equal to the maximum across all buckets! Pre-processing Rulesets To address this problem, we have the server perform a pre-processing step in which it attempts to balance buckets by generating a “ruleset”: a set of operations to process and hash a given URL. The server computes this ruleset and shares it with clients ahead of time so that the client can apply the same set of rules at lookup time. Here’s an example of a ruleset containing three rules: Hash Prefix # of Path Segments 08bd4dd11758b503 2 fe891588d205cf7f 1 c078e5ff2e262830 4 Each row is a rule that maps an 8-byte hash prefix to a certain number of path segments to append to the running URL query. Using our example of the link “example.com/a/b/index.html,” the client starts by computing a short hash of the domain: Hash(“example.com”). Let’s say that it matches one of the hashes in the ruleset, 08bd4dd11758b503. Then the client is instructed to recompute the hash after appending two path segments, meaning that the client computes the new hash as Hash(“example.com/a/b”) and again checks to see if the ruleset contains an entry for the new hash. The client repeats these steps until the hash prefix does not exist in the ruleset, at which point it stops and outputs the first two bytes of that hash prefix as a bucket identifier. The server generates the ruleset in an iterative process. The server starts with the assumption that each URL is hashed only by its domain and computes the initial buckets. It then identifies the largest bucket and finds the most common domain in that bucket. Then, it breaks up that bucket by adding a rule to append one or more additional URL segments for that domain. This process is repeated until all buckets are below an acceptable threshold. Because of the way the ruleset is generated, any URL that has a blocked prefix is guaranteed to hash to the bucket containing that entry. This invariant holds so long as the blocklist doesn’t contain redundant entries (e.g., one entry for “example.com” and another for “example.com/a”) and as long as the hash function used for ruleset mapping doesn’t produce any collisions among blocklist entries. At lookup time, the client uses the same ruleset to compute the URL’s bucket identifier. The client sends the bucket identifier to the server alongside an OPRF-blinded element for each path segment of the query link. The server responds with the bucket contents and the OPRF-blinded responses. Finally, the client unblinds the OPRF output and checks for an exact match of any of the OPRF outputs in the bucket contents. If a match is found, then the URL is flagged. Note that in order to hide the number of path segments of the query link from the server, we must appropriately pad up to a fixed maximum number of elements in order to prevent the length of the request from revealing information about the link. Likewise, we must also pad the bucket contents so that all buckets are of the same length, so that the length of the server response doesn’t reveal information about the client’s link. Safeguarding Client Queries Now, in the description of this protocol so far, the client still sends a bucket identifier (computed from the URL) to the server in order to be able to efficiently process the query. We can use additional mechanisms to further reduce the bits of information that a hypothetically adversarial server could glean from the client’s query, which we will cover in the following sections. Confidential Computing In order to limit the exposure of these hash prefixes to Meta’s servers, we leverage AMD’s SEV-SNP technology to provide a confidential virtual machine (CVM) for which the server-side code processes these hash prefixes. At a high level, the CVM provides an environment for us to run application code that we can generate attestation reports for. It also allows us to bootstrap a secure channel from a client to the CVM after the client establishes “trust” by verifying these attestation reports. An attestation report contains: A container manifest containing hash digests of the CVM’s launch configuration and packages, which essentially acts as a commitment to the application logic running on the CVM. A public key generated on CVM startup, corresponding to a private key that remains secured within the TEE. A certificate chain, with its root certificate established by AMD’s Key Distribution Service. A signature from the transparency log witness, which provides a uniqueness guarantee that mitigates server-side equivocation Upon receiving this report, the client verifies all of the certificates/signatures and then uses the embedded public key to establish a secure channel with the CVM. This secure channel is used by the client to transmit the bucket identifier to the CVM, which then uses the corresponding decryption key to decrypt the client’s request to obtain the plaintext bucket identifier. Last year, we posted about our usage of AMD SEV-SNP for providing a trusted execution environment for WhatsApp Private Processing, and many of the details behind the hardware setup are similar there. One aspect missing from this verification procedure is the releasing of these artifacts for external security researchers to be able to validate. We aim to provide a platform for hosting these artifacts in the near future. Oblivious RAM While the hardware guarantees provided by AMD SEV-SNP do allow us to reduce the exposure of these hash prefixes and send them through an encrypted channel, they are not sufficient by themselves to fully hide these hash prefixes from an observer that obtains administrative privileges of the host system to monitor memory accesses over time. Although the memory pages are encrypted through AMD’s Secure Nested Paging (SNP) technology, the patterns of access themselves must also be kept private. A straightforward way to address this would be to load the database into the machine’s memory on startup, and upon every client request, ensure that each one of the B buckets in the database is retrieved from memory, even though only one bucket is actually included in the server’s response. While this is fairly wasteful from a purely computational perspective (the B-1 accesses don’t actually factor into the response), the server can avoid directly leaking the bucket index being fetched to an adversary that can observe its memory access patterns when serving client requests. For a really large database, these B-1 accesses can end up being a bottleneck on the overall runtime of the server. There are two methods we leverage to optimize this performance overhead without compromising on privacy: Since our database is (at the time of writing) not overwhelmingly large, we can fit multiple disparate copies of the same database into memory on a single machine. Incoming client requests are assigned one of these copies based on availability, since the linear scan is inherently sequential in nature. We can improve on the number of accesses asymptotically, from linear to sublinear, by relying on an algorithm called Path ORAM. The exact details of how Path ORAM works in our setting is beyond the scope of this post, but you can find more information about this in our open-source library for Path ORAM. Using Oblivious HTTP To further strengthen the privacy guarantees of ABP, we leverage a third-party proxy and the Oblivious HTTP (OHTTP) protocol to de-identify client requests. The third-party proxy sits in between the client and server, processing encrypted client requests by stripping out identifying information from them and forwarding these de-identified requests to the server, which, in turn, is able to decrypt the request payload. This makes it more difficult for the server to be able to observe identifiers (such as the client’s IP address). The ABP Request Lifecycle The overall lifecycle of ABP for a request works as follows: Pre-processing/background phase: On a periodic basis, the server pulls in the latest updates to the URL database, iteratively computing a ruleset that balances the database entries into similarly-sized buckets. These buckets are then loaded onto a TEE using ORAM. The TEE generates a keypair, and the public key is embedded in an attestation report, generated by AMD SEV-SNP hardware. The attestation report and the current ruleset for the database are provided to the client upon request (through a third-party proxy). The client verifies the signatures contained in the attestation report, and locally stores a copy of the public key and database ruleset. And then, on each client request corresponding to a link click: The client, upon clicking a link in an E2EE chat, calculates the bucket identifier for the link by applying the rules of the “ruleset” to the URL. This bucket identifier is encrypted for the specific CVM instance using its public key. The client also computes a series of OPRF requests (blinded group elements), one for each path segment of the URL (padded). The encrypted bucket identifier, along with these OPRF requests, are sent through a third-party proxy to the server, along with a client public key as part of the establishment of a secure channel. The server precomputes the server-side evaluation of the OPRF requests to produce OPRF responses. The server then decrypts the bucket identifier, uses ORAM to look up the corresponding bucket contents, and returns the OPRF responses and bucket contents to the client, encrypted under the client’s public key. The client then decrypts the server’s response, and uses the bucket contents along with the OPRF responses to complete the OPRF evaluation and determine if a match was found. If a match was found, then the client displays a warning about the query link. The post How Advanced Browsing Protection Works in Messenger appeared first on Engineering at Meta.
FFmpeg at Meta: Media Processing at Scale
FFmpeg is truly a multi-tool for media processing. As an industry-standard tool it supports a wide variety of audio and video codecs and container formats. It can also orchestrate complex chains of filters for media editing and manipulation. For the people who use our apps, FFmpeg plays an important role in enabling new video experiences [...] Read More... The post FFmpeg at Meta: Media Processing at Scale appeared first on Engineering at Meta.
FFmpeg is truly a multi-tool for media processing. As an industry-standard tool it supports a wide variety of audio and video codecs and container formats. It can also orchestrate complex chains of filters for media editing and manipulation. For the people who use our apps, FFmpeg plays an important role in enabling new video experiences and improving the reliability of existing ones. Meta executes ffmpeg (the main CLI application) and ffprobe (a utility for obtaining media file properties) binaries tens of billions of times a day, introducing unique challenges when dealing with media files. FFmpeg can easily perform transcoding and editing on individual files, but our workflows have additional requirements to meet our needs. For many years we had to rely on our own internally developed fork of FFmpeg to provide features that have only recently been added to FFmpeg, such as threaded multi-lane encoding and real-time quality metric computation. Over time, our internal fork came to diverge significantly from the upstream version of FFmpeg. At the same time, new versions of FFmpeg brought support for new codecs and file formats, and reliability improvements, all of which allowed us to ingest more diverse video content from users without disruptions. This necessitated that we support both recent open-source versions of FFmpeg alongside our internal fork. Not only did this create a gradually divergent feature set, it also created challenges around safely rebasing our internal changes to avoid regressions. As our internal fork became increasingly outdated, we collaborated with FFmpeg developers, FFlabs, and VideoLAN to develop features in FFmpeg that allowed us to fully deprecate our internal fork and rely exclusively on the upstream version for our use cases. Using upstreamed patches and refactorings we’ve been able to fill two important gaps that we had previously relied on our internal fork to fill: threaded, multi-lane transcoding and real-time quality metrics. Building More Efficient Multi-Lane Transcoding for VOD and Livestreaming <img loading="lazy" decoding="async" class="size-full wp-image-23676" src="https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg" alt="" width="1920" height="1080" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg 1920w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=916,515 916w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-2.jpg?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> A video transcoding pipeline producing multiple outputs at different resolutions. When a user uploads a video through one of our apps, we generate a set of encodings to support Dynamic Adaptive Streaming over HTTP (DASH) playback. DASH playback allows the app’s video player to dynamically choose an encoding based on signals such as network conditions. These encodings can differ in resolution, codec, framerate, and visual quality level but they are created from the same source encoding, and the player can seamlessly switch between them in real time. In a very simple system separate FFmpeg command lines can generate the encodings for each lane one-by-one in serial. This could be optimized by running each command in parallel, but this quickly becomes inefficient due to the duplicate work done by each process. To work around this, multiple outputs could be generated within a single FFmpeg command line, decoding the frames of a video once and sending them to each output’s encoder instance. This eliminates a lot of overhead by deduplicating the video decoding and process startup time overhead incurred by each command line. Given that we process over 1 billion video uploads daily, each requiring multiple FFmpeg executions, reductions in per-process compute usage yield significant efficiency gains. Our internal FFmpeg fork provided an additional optimization to this: parallelized video encoding. While individual video encoders are often internally multi-threaded, previous FFmpeg versions executed each encoder in serial for a given frame when multiple encoders were in use. By running all encoder instances in parallel, better parallelism can be obtained overall. Thanks to contributions from FFmpeg developers, including those at FFlabs and VideoLAN, more efficient threading was implemented starting with FFmpeg 6.0, with the finishing touches landing in 8.0. This was directly influenced by the design of our internal fork and was one of the main features we had relied on it to provide. This development led to the most complex refactoring of FFmpeg in decades and has enabled more efficient encodings for all FFmpeg users. To fully migrate off of our internal fork we needed one more feature implemented upstream: real-time quality metrics. Enabling Real-Time Quality Metrics While Transcoding for Livestreams <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23675" src="https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg" alt="" width="1920" height="1080" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg 1920w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=916,515 916w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2026/02/FFmpeg-at-Meta-image-1.jpg?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Visual quality metrics, which give a numeric representation of the perceived visual quality of media, can be used to quantify the quality loss incurred from compression. These metrics are categorized as reference or no-reference metrics, where the former compares a reference encoding to some other distorted encoding. FFmpeg can compute various visual quality metrics such as PSNR, SSIM, and VMAF using two existing encodings in a separate command line after encoding has finished. This is okay for offline or VOD use cases, but not for livestreaming where we might want to compute quality metrics in real time. To do this, we need to insert a video decoder after each video encoder used by each output lane. These provide bitmaps for each frame in the video after compression has been applied so that we can compare against the frames before compression. In the end, we can produce a quality metric for each encoded lane in real time using a single FFmpeg command line. Thanks to “in-loop” decoding, which was enabled by FFmpeg developers including those from FFlabs and VideoLAN, beginning with FFmpeg 7.0, we no longer have to rely on our internal FFmpeg fork for this capability. We Upstream When It Will Have the Most Community Impact Things like real-time quality metrics while transcoding and more efficient threading can bring efficiency gains to a variety of FFmpeg-based pipelines both in and outside of Meta, and we strive to enable these developments upstream to benefit the FFmpeg community and wider industry. However, there are some patches we’ve developed internally that don’t make sense to contribute upstream. These are highly specific to our infrastructure and don’t generalize well. FFmpeg supports hardware-accelerated decoding, encoding, and filtering with devices such as NVIDIA’s NVDEC and NVENC, AMD’s Unified Video Decoder (UVD), and Intel’s Quick Sync Video (QSV). Each device is supported through an implementation of standard APIs in FFmpeg, allowing for easier integration and minimizing the need for device-specific command line flags. We’ve added support for the Meta Scalable Video Processor (MSVP), our custom ASIC for video transcoding, through these same APIs, enabling the use of common tooling across different hardware platforms with minimal platform-specific quirks. As MSVP is only used within Meta’s own infrastructure, it would create a challenge for FFmpeg developers to support it without access to the hardware for testing and validation. In this case, it makes sense to keep patches like this internal since they wouldn’t provide benefit externally. We’ve taken on the responsibility of rebasing our internal patches onto more recent FFmpeg versions over time, utilizing extensive validation to ensure robustness and correctness during upgrades. Our Continued Commitment to FFmpeg With more efficient multi-lane encoding and real-time quality metrics, we were able to fully deprecate our internal FFmpeg fork for all VOD and livestreaming pipelines. And thanks to standardized hardware APIs in FFmpeg, we’ve been able to support our MSVP ASIC alongside software-based pipelines with minimal friction. FFmpeg has withstood the test of time with over 25 years of active development. Developments that improve resource utilization, add support for new codecs and features, and increase reliability enable robust support for a wider range of media. For people on our platforms, this means enabling new experiences and improving the reliability of existing ones. We plan to continue investing in FFmpeg in partnership with open source developers, bringing benefits to Meta, the wider industry, and people who use our products. Acknowledgments We would like to acknowledge contributions from the open source community, our partners in FFlabs and VideoLAN, and many Meta engineers, including Max Bykov, Jordi Cenzano Ferret, Tim Harris, Colleen Henry, Mark Shwartzman, Haixia Shi, Cosmin Stejerean, Hassene Tmar, and Victor Loh. The post FFmpeg at Meta: Media Processing at Scale appeared first on Engineering at Meta.
Investing in Infrastructure: Meta’s Renewed Commitment to jemalloc
Meta recognizes the long-term benefits of jemalloc, a high-performance memory allocator, in its software infrastructure. We are renewing focus on jemalloc, aiming to reduce maintenance needs and modernize the codebase while continuing to evolve the allocator to adapt to the latest hardware and workloads. We are committed to continuing to develop jemalloc development with the [...] Read More... The post Investing in Infrastructure: Meta’s Renewed Commitment to jemalloc appeared first on Engineering at Meta.
Meta recognizes the long-term benefits of jemalloc, a high-performance memory allocator, in its software infrastructure. We are renewing focus on jemalloc, aiming to reduce maintenance needs and modernize the codebase while continuing to evolve the allocator to adapt to the latest hardware and workloads. We are committed to continuing to develop jemalloc development with the open source community and welcome contributions and collaborations from the community. Building a software system is a lot like building a skyscraper: The product everyone sees is the top, but the part that keeps it from falling over is the foundation buried in the dirt and the scaffolding hidden from sight. jemalloc, the high performance memory allocator, has consistently been a highly-leveraged component within our software stack, adapting over time to changes in underlying hardware and upper-layer software. Alongside the Linux kernel and the compilers, it has delivered long-term benefits to Meta, contributing to a reliable and performant infrastructure. Listening, Reflecting, and Changing High leverage comes with high stakes. On the spectrum of practical versus principled engineering practice, foundational software components like jemalloc need the highest rigor. With the leverage jemalloc provides however, it can be tempting to realize some short-term benefit. It requires strong self-discipline as an organization to resist that temptation and adhere to the core engineering principles. In recent years, there has been a gradual shift away from the core engineering principles that have long guided jemalloc’s development. While some decisions delivered immediate benefits, the resulting technical debt eventually slowed progress. We took the community’s feedback to heart. In the spirit of collaboration, we have reflected deeply on our stewardship and its impact on jemalloc’s long-term health. As we’ve met with some members of the community, including the project’s founder, Jason Evans, to share our introspection and how we are changing our approach. We’ve started an effort to remove technical debt and rebuild a long-term roadmap for jemalloc. A New Chapter for jemalloc As a result of these conversations with the community, the original jemalloc open source repository has been unarchived. We are grateful for the opportunity to continue as stewards of the project. Meta is renewing focus on jemalloc, aiming to reduce maintenance needs and modernizing the codebase while continuing to evolve the allocator to adapt to the latest and emerging hardware and workloads. Looking ahead, our current plan for jemalloc focus on several key areas of improvement: Technical Debt Reduction: We are focusing on cleaning up technical debt, refactoring, and enhancing jemalloc to ensure it remains efficient, reliable and easy to use for all users. Huge-Page Allocator: We will continue to improve jemalloc’s hugepage allocator (HPA) to better utilize transparent hugepages (THP) for improved CPU efficiency. Memory Efficiency: We plan to deliver improvements to packing, caching, and purging mechanisms for optimized memory efficiency. AArch64 Optimizations: We will make sure jemalloc has good out-of-the-box performance for the AArch64 (ARM64) platform. We know that trust is earned through action. Our hope is that, over time, our renewed commitment will be evident in the health and progress of jemalloc. We invite the community to join us in this new chapter — share your feedback and help shape jemalloc’s future. We look forward to collaborating with the community to drive jemalloc forward. The post Investing in Infrastructure: Meta’s Renewed Commitment to jemalloc appeared first on Engineering at Meta.
RCCLX: Innovating GPU Communications on AMD Platforms
We are open-sourcing the initial version of RCCLX – an enhanced version of RCCL that we developed and tested on Meta’s internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communication patterns for AI models are constantly evolving, as are hardware [...] Read More... The post RCCLX: Innovating GPU Communications on AMD Platforms appeared first on Engineering at Meta.
We are open-sourcing the initial version of RCCLX – an enhanced version of RCCL that we developed and tested on Meta’s internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communication patterns for AI models are constantly evolving, as are hardware capabilities. We want to iterate on collectives, transports, and novel features quickly on AMD platforms. Earlier, we developed and open-sourced CTran, a custom transport library on the NVIDIA platform. With RCCLX, we have integrated CTran to AMD platforms, enabling the AllToAllvDynamic – a GPU-resident collective. While not all the CTran features are currently integrated into the open source RCCLX library, we’re aiming to have them available in the coming months. In this post, we highlight two new features – Direct Data Access (DDA) and Low Precision Collectives. These features provide significant performance improvements on AMD platforms and we are excited to share this with the community. Direct Data Access (DDA) – Lightweight Intra-node Collectives Large language model inference operates through two distinct computational stages, each with fundamentally different performance characteristics: The prefill stage processes the input prompt, which can span thousands of tokens, to generate a key-value (KV) cache for each transformer layer of the model. This stage is compute-bound because the attention mechanism scales quadratically with sequence length, making it highly demanding on GPU computational resources. The decoding stage then utilizes and incrementally updates the KV cache to generate tokens one by one. Unlike prefill, decoding is memory-bound, as the I/O time of reading memory dominates attention time, with model weights and the KV cache occupying the majority of memory. Tensor parallelism enables models to be distributed across multiple GPUs by sharding individual layers into smaller, independent blocks that execute on different devices. However, one important challenge is the AllReduce communication operation can contribute up to 30% of end-to-end (E2E) latency. To address this bottleneck, Meta developed two DDA algorithms. The DDA flat algorithm improves small message-size allreduce latency by allowing each rank to directly load memory from other ranks and perform local reduce operations, reducing latency from O(N) to O(1) by increasing the data exchange from O(n) to O(n²). The DDA tree algorithm breaks the allreduce into two phases (reduce-scatter and all-gather) and uses direct data access in each step, moving the same amount of data as the ring algorithm but reducing latency to a constant factor for slightly larger message sizes. <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23620" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png" alt="" width="1442" height="780" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png 1442w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=916,495 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=768,415 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=1024,554 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=96,52 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=192,104 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23624" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png" alt="" width="1052" height="710" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png 1052w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=916,618 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=768,518 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=1024,691 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=96,65 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=192,130 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> The performance improvements of DDA over baseline communication libraries are substantial, particularly on AMD hardware. With AMD MI300X GPUs, DDA outperforms the RCCL baseline by 10-50% for decode (small message sizes) and yields 10-30% speedup for prefill. These improvements resulted in approximately 10% reduction in time-to-incremental-token (TTIT), directly enhancing the user experience during the critical decoding phase. Low-precision Collectives Low-precision (LP) collectives are a set of distributed communication algorithms — AllReduce, AllGather, AlltoAll, and ReduceScatter — optimized for AMD Instinct MI300/MI350 GPUs to accelerate AI training and inference workloads. These collectives support both FP32 and BF16 data types, leveraging FP8 quantization for up to 4:1 compression, which significantly reduces communication overhead and improves scalability and resource utilization for large message sizes (≥16MB). The algorithms use parallel peer-to-peer (P2P) mesh communication, fully exploiting AMD’s Infinity Fabric for high bandwidth and low latency, while compute steps are performed in high precision (FP32) to maintain numerical stability. Precision loss is primarily dictated by the number of quantization operations — typically one or two per data type in each collective — and whether the data can be adequately represented within the FP8 range. By dynamically enabling LP collectives, users can selectively activate these optimizations in E2E scenarios that benefit most from performance gains. Based on internal experiments, we have observed significant speed up for FP32 and notable improvements for BF16; it’s important to note that these collectives have been tuned for single-node deployments at this time. Reducing the precision of types can potentially have an impact on numeric accuracy so we tested for this and we found that it provided acceptable numerical accuracy for our workloads. This flexible approach allows teams to maximize throughput while maintaining acceptable numerical accuracy, and is now fully integrated and available in RCCLX for AMD platforms — simply set the environment variable RCCL_LOW_PRECISION_ENABLE=1 to get started. <img loading="lazy" decoding="async" class="size-full wp-image-23619" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png" alt="" width="1218" height="750" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png 1218w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=916,564 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=768,473 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=1024,631 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=192,118 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllReduce speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23621" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png" alt="" width="1214" height="746" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png 1214w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=916,563 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=768,472 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=1024,629 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=192,118 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllGather speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23623" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png" alt="" width="1216" height="742" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png 1216w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=916,559 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=768,469 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=1024,625 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=192,117 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllToAll speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23618" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png" alt="" width="1206" height="748" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png 1206w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=916,568 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=768,476 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=1024,635 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=96,60 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=192,119 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP ReduceScatter speedup. We are observing the following results from E2E inference workload evaluations when selectively enabling LP collectives: Approximately ~0.3% delta on GSM8K evaluation runs. ~9–10% decrease in latency. ~7% increase in throughput. The throughput measurements shown in the graphs were obtained using param-bench rccl-tests. For the MI300, the tests were run on RCCLX built with ROCm 6.4, and for the MI350, on RCCLX built with ROCm 7.0. Each test included 10 warmup iterations followed by 100 measurement iterations. The reported results represent the average throughput across the measurement iterations. Easy adaptation of AI models RCCLX is integrated with the Torchcomms API as a custom backend. We aim for this backend to have feature parity with our NCCLX backend (for NVIDIA platforms). Torchcomms allows users to have a single API for communication for different platforms. A user would not need to change the APIs they’re familiar with to port their applications across AMD, or other platforms even when using the novel features provided by CTran. <img loading="lazy" decoding="async" class="size-full wp-image-23625" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg" alt="" width="1072" height="342" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg 1072w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=916,292 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=768,245 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=1024,327 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=96,31 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=192,61 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23626" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg" alt="" width="1070" height="348" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg 1070w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=916,298 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=768,250 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=1024,333 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=96,31 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=192,62 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> RCCLX Quick Start Guide Install Torchcomms with RCCLX backend by following the installation instructions in the Torchcomms repo. import torchcomms # Eagerly initialize a communicator using MASTER_PORT/MASTER_ADDR/RANK/WORLD_SIZE environment variables provided by torchrun. # This communicator is bound to a single device. comm = torchcomms.new_comm("rcclx", torch.device("hip"), name="my_comm") print(f"I am rank {comm.get_rank()} of {comm.get_size()}!") t = torch.full((10, 20), value=comm.rank, dtype=torch.float) # run an all_reduce on the current stream comm.allreduce(t, torchcomms.ReduceOp.SUM, async_op=False) Acknowledgements We extend our gratitude to the AMD RCCL team for their ongoing collaboration. We also want to recognize the many current and former Meta employees whose contributions were vital in developing torchcomms and torchcomms-backends for production-scale training and inference. In particular, we would like to give special thanks to Dingming Wu, Qiye Tan, Pavan Balaji Yan Cui, Zhe Qu, Ahmed Khan, Ajit Mathews, CQ Tang, Srinivas Vaidyanathan, Harish Kumar Chandrappa, Peng Chen, Shashi Gandham, and Omar Baldonado The post RCCLX: Innovating GPU Communications on AMD Platforms appeared first on Engineering at Meta.
The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It
WHAT IT IS The rise of agentic software development means code is being written, reviewed, and shipped faster than ever before across the entire industry. It also means that testing frameworks need to evolve for this rapidly changing landscape. Faster development demands faster testing that can catch bugs as they land in a codebase, without [...] Read More... The post The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It appeared first on Engineering at Meta.
WHAT IT IS The rise of agentic software development means code is being written, reviewed, and shipped faster than ever before across the entire industry. It also means that testing frameworks need to evolve for this rapidly changing landscape. Faster development demands faster testing that can catch bugs as they land in a codebase, without requiring regular updates and maintenance. Just-in‑Time Tests (JiTTests) are a fundamentally novel approach to testing where tests are automatically generated by large language models (LLMs) on the fly to catch bugs – even ones that traditional testing might not catch – just-in-time before the code lands into production. A Catching JiTTest focuses specifically on finding regressions introduced by a code change. This type of testing reimagines decades of software testing theory and practice. While traditional testing relies on static test suites, manual authoring, and ongoing maintenance, Catching JiTTests require no test maintenance and no test code review, meaning engineers can focus their expertise on real bugs, not false positives. Catching JiTTests use sophisticated techniques to maximize test signal value and minimize false positive drag, targeting test signals where they matter most: on serious failures. HOW TESTING TRADITIONALLY WORKS Under the traditional paradigm, tests are manually built as new code lands in a codebase and continually executed, requiring regular updates and maintenance. The engineers building these tests face the challenge of needing to check the behavior, not only of the current code, but all possible future changes. Inherent uncertainty about future changes results in tests that don’t catch anything, or when they do, it’s a false positive. Agentic development dramatically increases the pace of code change, straining test development burden and scaling the cost of false positives and test maintenance to breaking point. HOW CATCHING JITTESTS WORK Broadly, JiTTests are bespoke tests, tailored to a specific code change, that give engineers simple, actionable feedback about unexpected behavior changes without the need to read or write test code. LLMs can generate JiTTests automatically the moment a pull request is submitted. And since the JiTTest itself is LLM-generated, it can often infer the plausible intention of a code change and simulate possible faults that may result from it. With an understanding of intent, Catching JiTTests can significantly drive down instances of false positives. Here are the key steps of the Catching JiTTest process: New code lands in the codebase. The system infers the intention of the code change. It creates mutants (code versions with faults deliberately inserted) to simulate what could go wrong. It generates and runs tests to catch those faults. Ensembles of rule-based and LLM-based assessors focus the signal on true positive failures. Engineers receive clear, relevant reports about unexpected changes right when it matters most. WHY IT MATTERS Catching JiTTests are designed for the world of AI-powered agentic software development and accelerate testing by focusing on serious unexpected bugs. With them engineers no longer have to spend time writing, reviewing, and testing complex test code. Catching JiTTests, by design, kill many of the issues with traditional testing in one stroke: They are generated on-the-fly for each code change and do not reside in the codebase, eliminating ongoing maintenance costs and shifting effort from humans to machines. They are tailored to each change, making them more robust and less prone to breaking due to intended updates. They automatically adapt as the code changes. They only require human review when a bug is actually caught. This all amounts to an important shift in testing infrastructure where the focus moves from generic code quality to whether a test actually finds faults in a specific change without raising a false positive. It helps improve testing overall while also allowing it to keep up with the pace of agentic coding. READ THE PAPER Just-in-Time Catching Test Generation at Meta The post The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It appeared first on Engineering at Meta.
Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters
We’re sharing details of the role backend aggregation (BAG) plays in building Meta’s gigawatt-scale AI clusters like Prometheus. BAG allows us to seamlessly connect thousands of GPUs across multiple data centers and regions. Our BAG implementation is connecting two different network fabrics – Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF). Once it’s complete our AI [...] Read More... The post Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters appeared first on Engineering at Meta.
We’re sharing details of the role backend aggregation (BAG) plays in building Meta’s gigawatt-scale AI clusters like Prometheus. BAG allows us to seamlessly connect thousands of GPUs across multiple data centers and regions. Our BAG implementation is connecting two different network fabrics – Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF). Once it’s complete our AI cluster, Prometheus, will deliver 1-gigawatt of capacity to enhance and enable new and existing AI experiences across Meta products. Prometheus’ infrastructure will span several data center buildings in a single larger region, interconnecting tens of thousands of GPUs. A key piece of scaling and connecting this infrastructure is backend aggregation (BAG), which we use to seamlessly connect GPUs and data centers with robust, high-capacity networking. By leveraging modular hardware, advanced routing, and resilient topologies, BAG ensures both performance and reliability at unprecedented scale As our AI clusters continue to grow, we expect BAG to play an important role in meeting future demands and driving innovation across Meta’s global network. What Is Backend Aggregation? BAG is a centralized Ethernet-based super spine network layer that primarily functions to interconnect multiple spine layer fabrics across various data centers and regions within large clusters. Within Prometheus, for example, the BAG layer serves as the aggregation point between regional networks and Meta’s backbone, enabling the creation of mega AI clusters. BAG is designed to support immense bandwidth needs, with inter-BAG capacities reaching the petabit range (e.g., 16-48 Pbps per region pair). <img loading="lazy" decoding="async" class="size-full wp-image-23636" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png" alt="" width="1999" height="970" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png 1999w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=916,444 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=768,373 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=1024,497 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=1536,745 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=96,47 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=192,93 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> We use backend aggregation (BAG) to interconnect data center regions to share compute and other resources into large clusters. How BAG Is Helping Us Build Gigawatt-Scale AI Clusters To address the challenge of interconnecting tens of thousands of GPUs, we’re deploying distributed BAG layers regionally. How We Interconnect BAG Layers BAG layers are strategically distributed across regions to serve subsets of L2 fabrics, adhering to distance, buffer, and latency constraints. Inter-BAG connectivity utilizes either a planar (direct match) or spread connection topology, chosen based on site size and fiber availability. Planar topology connects BAG switches one-to-one between regions following the plane, offering simplified management but concentrating potential failure domains. Spread connection topology distributes links across multiple BAG switches/planes, enhancing path diversity and resilience. <img loading="lazy" decoding="async" class="size-full wp-image-23638" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png" alt="" width="1480" height="1180" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png 1480w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=916,730 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=768,612 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=1024,816 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=96,77 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=192,153 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> An example of an inter-BAG network topology. How a BAG Layer Connects to L2 Fabrics So far, we’ve discussed how the BAG layers are interconnected, now let’s see how a BAG layer connects downstream to L2 fabrics. We’ve used two main fabric technologies, Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF) to build L2 networks. Below is an example of DSF L2 zones across five data center buildings connected to the BAG layer via a special backend edge pod in each building. <img loading="lazy" decoding="async" class="size-full wp-image-23637" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png" alt="" width="1842" height="952" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png 1842w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=916,473 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=768,397 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=1024,529 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=1536,794 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=96,50 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=192,99 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> A BAG inter-building connection for DSF fabric across five data centers. Below is an example of NSF L2 connected to BAG planes. Each BAG plane connects to matching Spine Training Switches (STSWs) from all spine planes. Effective oversubscription is 4.98:1. <img loading="lazy" decoding="async" class="size-full wp-image-23635" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png" alt="" width="1600" height="870" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png 1600w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=916,498 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=768,418 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=1024,557 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=1536,835 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=96,52 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=192,104 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> A BAG inter-building connection for NSF fabric. Careful management of oversubscription ratios assists in balancing scale and performance. Typical oversubscription from L2 to BAG is around 4.5:1, while BAG-to-BAG oversubscription varies based on regional requirements and link capacity. Hardware and Routing Meta’s implementation of BAG uses a modular chassis equipped with Jericho3 (J3) ASIC line cards, each providing up to 432x800G ports for high-capacity, scalable, and resilient interconnect. The central hub BAG employs a larger chassis to accommodate numerous spokes and long-distance links with varied cable lengths for optimized buffer utilization. Routing within BAG uses eBGP with link bandwidth attributes, enabling Unequal Cost Multipath (UCMP) for efficient load balancing and robust failure handling. BAG-to-BAG connections are secured with MACsec, aligning with network security requirements. Designing the Network for Resilience The network design meticulously details port striping, IP addressing schemes, and comprehensive failure domain analysis to ensure high availability and minimize the impact of failures. Failure modes are analyzed at the BAG, data hall, and power distribution levels. We also employ various strategies to mitigate blackholing risks, including draining affected BAG planes and conditional route aggregation. Considerations for Long Cable Distances An important advantage of BAG’s distributed architecture is it keeps the distance from the L2 edge small, which is important for shallow buffer NSF switches. Longer, BAG-to-BAG, cable distances dictate that we use deep buffer switches for the BAG role. This provides a large headroom buffer to support lossless congestion control protocols like PFC. Building Prometheus and Beyond As a technology, BAG is playing an important role in Meta’s next generation of AI infrastructure. By centralizing the interconnection of regional networks, BAG helps enable the gigawatt-scale Prometheus cluster, ensuring seamless, high-capacity networking across tens of thousands of GPUs. This thoughtful design, leveraging modular hardware and resilient topologies, positions BAG to not only meet the demands of Prometheus but also to drive the future innovation and scalability of Meta’s global AI network for years to come. The post Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters appeared first on Engineering at Meta.