RCCLX: Innovating GPU Communications on AMD Platforms
We are open-sourcing the initial version of RCCLX – an enhanced version of RCCL that we developed and tested on Meta’s internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communication patterns for AI models are constantly evolving, as are hardware [...] Read More... The post RCCLX: Innovating GPU Communications on AMD Platforms appeared first on Engineering at Meta.
We are open-sourcing the initial version of RCCLX – an enhanced version of RCCL that we developed and tested on Meta’s internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communication patterns for AI models are constantly evolving, as are hardware capabilities. We want to iterate on collectives, transports, and novel features quickly on AMD platforms. Earlier, we developed and open-sourced CTran, a custom transport library on the NVIDIA platform. With RCCLX, we have integrated CTran to AMD platforms, enabling the AllToAllvDynamic – a GPU-resident collective. While not all the CTran features are currently integrated into the open source RCCLX library, we’re aiming to have them available in the coming months. In this post, we highlight two new features – Direct Data Access (DDA) and Low Precision Collectives. These features provide significant performance improvements on AMD platforms and we are excited to share this with the community. Direct Data Access (DDA) – Lightweight Intra-node Collectives Large language model inference operates through two distinct computational stages, each with fundamentally different performance characteristics: The prefill stage processes the input prompt, which can span thousands of tokens, to generate a key-value (KV) cache for each transformer layer of the model. This stage is compute-bound because the attention mechanism scales quadratically with sequence length, making it highly demanding on GPU computational resources. The decoding stage then utilizes and incrementally updates the KV cache to generate tokens one by one. Unlike prefill, decoding is memory-bound, as the I/O time of reading memory dominates attention time, with model weights and the KV cache occupying the majority of memory. Tensor parallelism enables models to be distributed across multiple GPUs by sharding individual layers into smaller, independent blocks that execute on different devices. However, one important challenge is the AllReduce communication operation can contribute up to 30% of end-to-end (E2E) latency. To address this bottleneck, Meta developed two DDA algorithms. The DDA flat algorithm improves small message-size allreduce latency by allowing each rank to directly load memory from other ranks and perform local reduce operations, reducing latency from O(N) to O(1) by increasing the data exchange from O(n) to O(n²). The DDA tree algorithm breaks the allreduce into two phases (reduce-scatter and all-gather) and uses direct data access in each step, moving the same amount of data as the ring algorithm but reducing latency to a constant factor for slightly larger message sizes. <img fetchpriority="high" decoding="async" class="alignnone size-full wp-image-23620" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png" alt="" width="1442" height="780" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png 1442w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=916,495 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=768,415 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=1024,554 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=96,52 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-1.png?resize=192,104 192w" sizes="(max-width: 992px) 100vw, 62vw" /> <img decoding="async" class="alignnone size-full wp-image-23624" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png" alt="" width="1052" height="710" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png 1052w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=916,618 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=768,518 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=1024,691 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=96,65 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-.png?resize=192,130 192w" sizes="(max-width: 992px) 100vw, 62vw" /> The performance improvements of DDA over baseline communication libraries are substantial, particularly on AMD hardware. With AMD MI300X GPUs, DDA outperforms the RCCL baseline by 10-50% for decode (small message sizes) and yields 10-30% speedup for prefill. These improvements resulted in approximately 10% reduction in time-to-incremental-token (TTIT), directly enhancing the user experience during the critical decoding phase. Low-precision Collectives Low-precision (LP) collectives are a set of distributed communication algorithms — AllReduce, AllGather, AlltoAll, and ReduceScatter — optimized for AMD Instinct MI300/MI350 GPUs to accelerate AI training and inference workloads. These collectives support both FP32 and BF16 data types, leveraging FP8 quantization for up to 4:1 compression, which significantly reduces communication overhead and improves scalability and resource utilization for large message sizes (≥16MB). The algorithms use parallel peer-to-peer (P2P) mesh communication, fully exploiting AMD’s Infinity Fabric for high bandwidth and low latency, while compute steps are performed in high precision (FP32) to maintain numerical stability. Precision loss is primarily dictated by the number of quantization operations — typically one or two per data type in each collective — and whether the data can be adequately represented within the FP8 range. By dynamically enabling LP collectives, users can selectively activate these optimizations in E2E scenarios that benefit most from performance gains. Based on internal experiments, we have observed significant speed up for FP32 and notable improvements for BF16; it’s important to note that these collectives have been tuned for single-node deployments at this time. Reducing the precision of types can potentially have an impact on numeric accuracy so we tested for this and we found that it provided acceptable numerical accuracy for our workloads. This flexible approach allows teams to maximize throughput while maintaining acceptable numerical accuracy, and is now fully integrated and available in RCCLX for AMD platforms — simply set the environment variable RCCL_LOW_PRECISION_ENABLE=1 to get started. <img decoding="async" class="size-full wp-image-23619" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png" alt="" width="1218" height="750" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png 1218w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=916,564 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=768,473 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=1024,631 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-3.png?resize=192,118 192w" sizes="(max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllReduce speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23621" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png" alt="" width="1214" height="746" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png 1214w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=916,563 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=768,472 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=1024,629 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-4.png?resize=192,118 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllGather speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23623" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png" alt="" width="1216" height="742" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png 1216w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=916,559 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=768,469 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=1024,625 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=96,59 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-5.png?resize=192,117 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP AllToAll speedup. <img loading="lazy" decoding="async" class="size-full wp-image-23618" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png" alt="" width="1206" height="748" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png 1206w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=916,568 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=768,476 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=1024,635 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=96,60 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-6.png?resize=192,119 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> MI300 – Float LP ReduceScatter speedup. We are observing the following results from E2E inference workload evaluations when selectively enabling LP collectives: Approximately ~0.3% delta on GSM8K evaluation runs. ~9–10% decrease in latency. ~7% increase in throughput. The throughput measurements shown in the graphs were obtained using param-bench rccl-tests. For the MI300, the tests were run on RCCLX built with ROCm 6.4, and for the MI350, on RCCLX built with ROCm 7.0. Each test included 10 warmup iterations followed by 100 measurement iterations. The reported results represent the average throughput across the measurement iterations. Easy adaptation of AI models RCCLX is integrated with the Torchcomms API as a custom backend. We aim for this backend to have feature parity with our NCCLX backend (for NVIDIA platforms). Torchcomms allows users to have a single API for communication for different platforms. A user would not need to change the APIs they’re familiar with to port their applications across AMD, or other platforms even when using the novel features provided by CTran. <img loading="lazy" decoding="async" class="size-full wp-image-23625" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg" alt="" width="1072" height="342" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg 1072w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=916,292 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=768,245 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=1024,327 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=96,31 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-8.jpg?resize=192,61 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23626" src="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg" alt="" width="1070" height="348" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg 1070w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=916,298 916w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=768,250 768w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=1024,333 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=96,31 96w, https://engineering.fb.com/wp-content/uploads/2026/02/rrcclx-innovating-gpu-communications-amd-platforms-meta-image-7.jpg?resize=192,62 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> RCCLX Quick Start Guide Install Torchcomms with RCCLX backend by following the installation instructions in the Torchcomms repo. import torchcomms # Eagerly initialize a communicator using MASTER_PORT/MASTER_ADDR/RANK/WORLD_SIZE environment variables provided by torchrun. # This communicator is bound to a single device. comm = torchcomms.new_comm("rcclx", torch.device("hip"), name="my_comm") print(f"I am rank {comm.get_rank()} of {comm.get_size()}!") t = torch.full((10, 20), value=comm.rank, dtype=torch.float) # run an all_reduce on the current stream comm.allreduce(t, torchcomms.ReduceOp.SUM, async_op=False) Acknowledgements We extend our gratitude to the AMD RCCL team for their ongoing collaboration. We also want to recognize the many current and former Meta employees whose contributions were vital in developing torchcomms and torchcomms-backends for production-scale training and inference. In particular, we would like to give special thanks to Dingming Wu, Qiye Tan, Pavan Balaji Yan Cui, Zhe Qu, Ahmed Khan, Ajit Mathews, CQ Tang, Srinivas Vaidyanathan, Harish Kumar Chandrappa, Peng Chen, Shashi Gandham, and Omar Baldonado The post RCCLX: Innovating GPU Communications on AMD Platforms appeared first on Engineering at Meta.
The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It
WHAT IT IS The rise of agentic software development means code is being written, reviewed, and shipped faster than ever before across the entire industry. It also means that testing frameworks need to evolve for this rapidly changing landscape. Faster development demands faster testing that can catch bugs as they land in a codebase, without [...] Read More... The post The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It appeared first on Engineering at Meta.
WHAT IT IS The rise of agentic software development means code is being written, reviewed, and shipped faster than ever before across the entire industry. It also means that testing frameworks need to evolve for this rapidly changing landscape. Faster development demands faster testing that can catch bugs as they land in a codebase, without requiring regular updates and maintenance. Just-in‑Time Tests (JiTTests) are a fundamentally novel approach to testing where tests are automatically generated by large language models (LLMs) on the fly to catch bugs – even ones that traditional testing might not catch – just-in-time before the code lands into production. A Catching JiTTest focuses specifically on finding regressions introduced by a code change. This type of testing reimagines decades of software testing theory and practice. While traditional testing relies on static test suites, manual authoring, and ongoing maintenance, Catching JiTTests require no test maintenance and no test code review, meaning engineers can focus their expertise on real bugs, not false positives. Catching JiTTests use sophisticated techniques to maximize test signal value and minimize false positive drag, targeting test signals where they matter most: on serious failures. HOW TESTING TRADITIONALLY WORKS Under the traditional paradigm, tests are manually built as new code lands in a codebase and continually executed, requiring regular updates and maintenance. The engineers building these tests face the challenge of needing to check the behavior, not only of the current code, but all possible future changes. Inherent uncertainty about future changes results in tests that don’t catch anything, or when they do, it’s a false positive. Agentic development dramatically increases the pace of code change, straining test development burden and scaling the cost of false positives and test maintenance to breaking point. HOW CATCHING JITTESTS WORK Broadly, JiTTests are bespoke tests, tailored to a specific code change, that give engineers simple, actionable feedback about unexpected behavior changes without the need to read or write test code. LLMs can generate JiTTests automatically the moment a pull request is submitted. And since the JiTTest itself is LLM-generated, it can often infer the plausible intention of a code change and simulate possible faults that may result from it. With an understanding of intent, Catching JiTTests can significantly drive down instances of false positives. Here are the key steps of the Catching JiTTest process: New code lands in the codebase. The system infers the intention of the code change. It creates mutants (code versions with faults deliberately inserted) to simulate what could go wrong. It generates and runs tests to catch those faults. Ensembles of rule-based and LLM-based assessors focus the signal on true positive failures. Engineers receive clear, relevant reports about unexpected changes right when it matters most. WHY IT MATTERS Catching JiTTests are designed for the world of AI-powered agentic software development and accelerate testing by focusing on serious unexpected bugs. With them engineers no longer have to spend time writing, reviewing, and testing complex test code. Catching JiTTests, by design, kill many of the issues with traditional testing in one stroke: They are generated on-the-fly for each code change and do not reside in the codebase, eliminating ongoing maintenance costs and shifting effort from humans to machines. They are tailored to each change, making them more robust and less prone to breaking due to intended updates. They automatically adapt as the code changes. They only require human review when a bug is actually caught. This all amounts to an important shift in testing infrastructure where the focus moves from generic code quality to whether a test actually finds faults in a specific change without raising a false positive. It helps improve testing overall while also allowing it to keep up with the pace of agentic coding. READ THE PAPER Just-in-Time Catching Test Generation at Meta The post The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It appeared first on Engineering at Meta.
Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters
We’re sharing details of the role backend aggregation (BAG) plays in building Meta’s gigawatt-scale AI clusters like Prometheus. BAG allows us to seamlessly connect thousands of GPUs across multiple data centers and regions. Our BAG implementation is connecting two different network fabrics – Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF). Once it’s complete our AI [...] Read More... The post Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters appeared first on Engineering at Meta.
We’re sharing details of the role backend aggregation (BAG) plays in building Meta’s gigawatt-scale AI clusters like Prometheus. BAG allows us to seamlessly connect thousands of GPUs across multiple data centers and regions. Our BAG implementation is connecting two different network fabrics – Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF). Once it’s complete our AI cluster, Prometheus, will deliver 1-gigawatt of capacity to enhance and enable new and existing AI experiences across Meta products. Prometheus’ infrastructure will span several data center buildings in a single larger region, interconnecting tens of thousands of GPUs. A key piece of scaling and connecting this infrastructure is backend aggregation (BAG), which we use to seamlessly connect GPUs and data centers with robust, high-capacity networking. By leveraging modular hardware, advanced routing, and resilient topologies, BAG ensures both performance and reliability at unprecedented scale As our AI clusters continue to grow, we expect BAG to play an important role in meeting future demands and driving innovation across Meta’s global network. What Is Backend Aggregation? BAG is a centralized Ethernet-based super spine network layer that primarily functions to interconnect multiple spine layer fabrics across various data centers and regions within large clusters. Within Prometheus, for example, the BAG layer serves as the aggregation point between regional networks and Meta’s backbone, enabling the creation of mega AI clusters. BAG is designed to support immense bandwidth needs, with inter-BAG capacities reaching the petabit range (e.g., 16-48 Pbps per region pair). <img loading="lazy" decoding="async" class="size-full wp-image-23636" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png" alt="" width="1999" height="970" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png 1999w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=916,444 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=768,373 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=1024,497 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=1536,745 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=96,47 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-1.png?resize=192,93 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> We use backend aggregation (BAG) to interconnect data center regions to share compute and other resources into large clusters. How BAG Is Helping Us Build Gigawatt-Scale AI Clusters To address the challenge of interconnecting tens of thousands of GPUs, we’re deploying distributed BAG layers regionally. How We Interconnect BAG Layers BAG layers are strategically distributed across regions to serve subsets of L2 fabrics, adhering to distance, buffer, and latency constraints. Inter-BAG connectivity utilizes either a planar (direct match) or spread connection topology, chosen based on site size and fiber availability. Planar topology connects BAG switches one-to-one between regions following the plane, offering simplified management but concentrating potential failure domains. Spread connection topology distributes links across multiple BAG switches/planes, enhancing path diversity and resilience. <img loading="lazy" decoding="async" class="size-full wp-image-23638" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png" alt="" width="1480" height="1180" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png 1480w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=916,730 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=768,612 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=1024,816 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=96,77 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-2.png?resize=192,153 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> An example of an inter-BAG network topology. How a BAG Layer Connects to L2 Fabrics So far, we’ve discussed how the BAG layers are interconnected, now let’s see how a BAG layer connects downstream to L2 fabrics. We’ve used two main fabric technologies, Disaggregated Schedule Fabric (DSF) and Non-Scheduled Fabric (NSF) to build L2 networks. Below is an example of DSF L2 zones across five data center buildings connected to the BAG layer via a special backend edge pod in each building. <img loading="lazy" decoding="async" class="size-full wp-image-23637" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png" alt="" width="1842" height="952" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png 1842w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=916,473 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=768,397 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=1024,529 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=1536,794 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=96,50 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-3.png?resize=192,99 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> A BAG inter-building connection for DSF fabric across five data centers. Below is an example of NSF L2 connected to BAG planes. Each BAG plane connects to matching Spine Training Switches (STSWs) from all spine planes. Effective oversubscription is 4.98:1. <img loading="lazy" decoding="async" class="size-full wp-image-23635" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png" alt="" width="1600" height="870" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png 1600w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=916,498 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=768,418 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=1024,557 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=1536,835 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=96,52 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-Prometheus-Backend-Aggregation-BAG-image-4.png?resize=192,104 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> A BAG inter-building connection for NSF fabric. Careful management of oversubscription ratios assists in balancing scale and performance. Typical oversubscription from L2 to BAG is around 4.5:1, while BAG-to-BAG oversubscription varies based on regional requirements and link capacity. Hardware and Routing Meta’s implementation of BAG uses a modular chassis equipped with Jericho3 (J3) ASIC line cards, each providing up to 432x800G ports for high-capacity, scalable, and resilient interconnect. The central hub BAG employs a larger chassis to accommodate numerous spokes and long-distance links with varied cable lengths for optimized buffer utilization. Routing within BAG uses eBGP with link bandwidth attributes, enabling Unequal Cost Multipath (UCMP) for efficient load balancing and robust failure handling. BAG-to-BAG connections are secured with MACsec, aligning with network security requirements. Designing the Network for Resilience The network design meticulously details port striping, IP addressing schemes, and comprehensive failure domain analysis to ensure high availability and minimize the impact of failures. Failure modes are analyzed at the BAG, data hall, and power distribution levels. We also employ various strategies to mitigate blackholing risks, including draining affected BAG planes and conditional route aggregation. Considerations for Long Cable Distances An important advantage of BAG’s distributed architecture is it keeps the distance from the L2 edge small, which is important for shallow buffer NSF switches. Longer, BAG-to-BAG, cable distances dictate that we use deep buffer switches for the BAG role. This provides a large headroom buffer to support lossless congestion control protocols like PFC. Building Prometheus and Beyond As a technology, BAG is playing an important role in Meta’s next generation of AI infrastructure. By centralizing the interconnection of regional networks, BAG helps enable the gigawatt-scale Prometheus cluster, ensuring seamless, high-capacity networking across tens of thousands of GPUs. This thoughtful design, leveraging modular hardware and resilient topologies, positions BAG to not only meet the demands of Prometheus but also to drive the future innovation and scalability of Meta’s global AI network for years to come. The post Building Prometheus: How Backend Aggregation Enables Gigawatt-Scale AI Clusters appeared first on Engineering at Meta.
No Display? No Problem: Cross-Device Passkey Authentication for XR Devices
We’re sharing a novel approach to enabling cross-device passkey authentication for devices with inaccessible displays (like XR devices). Our approach bypasses the use of QR codes and enables cross-device authentication without the need for an on-device display, while still complying with all trust and proximity requirements. This approach builds on work done by the FIDO [...] Read More... The post No Display? No Problem: Cross-Device Passkey Authentication for XR Devices appeared first on Engineering at Meta.
We’re sharing a novel approach to enabling cross-device passkey authentication for devices with inaccessible displays (like XR devices). Our approach bypasses the use of QR codes and enables cross-device authentication without the need for an on-device display, while still complying with all trust and proximity requirements. This approach builds on work done by the FIDO Alliance and we hope it will open the door to bring secure, passwordless authentication to a whole new ecosystem of devices and platforms. Passkeys are a significant leap forward in authentication, offering a phishing-resistant, cryptographically secure alternative to traditional passwords. Generally, the standard cross-device passkey flow, where someone registers or authenticates on a desktop device by approving the action on their nearby mobile device, is done in a familiar way with QR codes scanned by their phone camera. But how can we facilitate this flow for XR devices with a head-mounted display or no screen at all, or for other devices with an inaccessible display like smart home hubs and industrial sensors? We’ve taken a novel approach to adapting the WebAuthn passkey flow and FIDO’s CTAP hybrid protocol for this unique class of devices that either lack a screen entirely or whose screen is not easily accessible to another device’s camera. Our implementation has been developed and is now broadly available on Meta Quest devices powered by Meta Horizon OS. We hope that this approach can also ensure robust security built on the strength of existing passkey frameworks, without sacrificing usability, for users of a variety of other screenless IoT devices, consumer electronics, and industrial hardware. The Challenge: No Screen, No QR Code The standard cross-device flow relies on two primary mechanisms: QR code scanning: The relying party displays a QR code on the desktop/inaccessible device, which the mobile authenticator scans to establish a secure link. Bluetooth/NFC proximity: The devices use local communication protocols to discover each other and initiate the secure exchange. For devices with no display, the QR code method is impossible. Proximity-based discovery is feasible, but initiating the user verification step and confirming the intent without any on-device visual feedback can introduce security and usability risks. People need clear assurance that they are approving the correct transaction on the correct device. Our Solution: Using a Companion App for Secure Message Transport Scanning a QR code sends the authenticator device a command to initiate a hybrid (cross-device) login flow with a nonce that identifies the unauthenticated device client. But if a user has a companion application – like the Meta Horizon app – that uses the same account as the device we can use that application to pass this same request to the authenticator OS and execute it using general link/intent execution. We made the flow easy to navigate by using in-app notifications to show users when a login request has been initiated, take them directly into the application, and immediately execute the login request. For simplicity, we opted to begin the hybrid flow as soon as the application is opened since the user would have had to take some action (clicking the notification or opening the app) to trigger this and there is an additional user verification step in hybrid implementations on iOS and Android. Here’s how this plays out on a Meta Quest with the Meta Horizon mobile app: <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23601" src="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png" alt="" width="1999" height="726" srcset="https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png 1999w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=916,333 916w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=768,279 768w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=1024,372 1024w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=1536,558 1536w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=96,35 96w, https://engineering.fb.com/wp-content/uploads/2026/02/Meta-XR-passkeys-flow.png?resize=192,70 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> 1. The Hybrid Flow Message Is Generated When a passkey login is initiated on the Meta Quest, the headset’s browser locally constructs the same payload that would have been embedded in a QR Code – including a fresh ECDH public key, a session-specific secret, and routing information used later in the handshake. Instead of rendering this information into an image (QR code), the browser encodes it into a FIDO URL (the standard mechanism defined for hybrid transport) that instructs the mobile device to begin the passkey authentication flow. 2. The Message Is Sent to the Companion App After the FIDO URL is generated, the headset requires a secure and deterministic method for transferring it to the user’s phone. Because the device cannot present a QR code, the system leverages the Meta Horizon app’s authenticated push channel to deliver the FIDO URL directly to the mobile device. When the user selects the passkey option in the login dialog, the headset encodes the FIDO URL as structured data within a GraphQL-based push notification. The Meta Horizon app, signed in with the same account as the headset, receives this payload and validates the delivery context to ensure it is routed to the correct user. 3. The Application Sends a Notification of the Login Request After the FIDO URL is delivered to the mobile device, the platform’s push service surfaces it as a standard iOS or Android notification indicating that a login request is pending. When the user taps the notification, the operating system routes the deep link to the Meta Horizon app. The app then opens the FIDO URL using the system URL launcher and invokes the operating system passkey interface. For users that have the notification turned off or disabled, launching the Meta Horizon app directly will also trigger a query to the backend for any pending passkey requests associated with the user’s account. If a valid request exists (requests expire after five minutes), the app automatically initiates the same passkey flow by opening the FIDO URL. Once the FIDO URL is opened, the mobile device begins the hybrid transport sequence, including broadcasting the BLE advertisement, establishing the encrypted tunnel, and producing the passkey assertion. In this flow, the system notification and the app launch path both serve as user consent surfaces and entry points into the standard hybrid transport workflow. 4. The App Executes the Hybrid Command Once the user approves the action on their mobile device, the secure channel is established as per WebAuthn standards. The main difference is the challenge exchange timing: The inaccessible device generates the standard WebAuthn challenge and waits. The mobile authenticator, initiates the secure BLE/NFC connection. The challenge is transmitted over this secure channel. Upon UV success, the mobile device uses the relevant key material to generate the AuthenticatorAssertionResponse or AuthenticatorAttestationResponse. The response is sent back to the inaccessible device. The inaccessible device then acts as the conduit, forwarding the response to the relying party server to complete the transaction, exactly as a standard display-equipped device would. Impact and Future Direction This novel implementation successfully bypasses the need for an on-device display in the cross-device flow and still complies with the proximity and other trust challenges that exist today for cross-device passkey login. We hope that our solution paves the way for secure, passwordless authentication across a wider range of different platforms and ecosystems, moving passkeys beyond just mobile and desktop environments and into the burgeoning world of wearable and IoT devices. We are proud to build on top of the excellent work already done in this area by our peers in the FIDO Alliance and mobile operating systems committed to this work and building a robust and interoperable ecosystem for secure and easy login. The post No Display? No Problem: Cross-Device Passkey Authentication for XR Devices appeared first on Engineering at Meta.
Rust at Scale: An Added Layer of Security for WhatsApp
WhatsApp has adopted and rolled out a new layer of security for users – built with Rust – as part of its effort to harden defenses against malware threats. WhatsApp’s experience creating and distributing our media consistency library in Rust to billions of devices and browsers proves Rust is production ready at a global scale. [...] Read More... The post Rust at Scale: An Added Layer of Security for WhatsApp appeared first on Engineering at Meta.
WhatsApp has adopted and rolled out a new layer of security for users – built with Rust – as part of its effort to harden defenses against malware threats. WhatsApp’s experience creating and distributing our media consistency library in Rust to billions of devices and browsers proves Rust is production ready at a global scale. Our Media Handling Strategy WhatsApp provides default end-to-end encryption for over 3 billion people to message securely each and every day. Online security is an adversarial space, and to continue ensuring users can keep messaging securely, we’re constantly adapting and evolving our strategy against cyber-security threats – all while supporting the WhatsApp infrastructure to help people connect. For example, WhatsApp, like many other applications, allows users to share media and other types of documents. WhatsApp helps protect users by warning about dangerous attachments like APKs, yet rare and sophisticated malware could be hidden within a seemingly benign file like an image or video. These maliciously crafted files might target unpatched vulnerabilities in the operating system, libraries distributed by the operating system, or the application itself. To help protect against such potential threads, WhatsApp is increasingly using the Rust programming language, including in our media sharing functionality. Rust is a memory safe language offering numerous security benefits. We believe that this is the largest rollout globally of any library written in Rust. To help explain why and how we rolled this out, we should first look back at a key OS-level vulnerability that sent an important signal to WhatsApp around hardening media-sharing defenses. 2015 Android Vulnerability: A Wake-up Call for Media File Protections In 2015, Android devices, and the applications that ran on them, became vulnerable to the “Stagefright” vulnerability. The bug lay in the processing of media files by operating system-provided libraries, so WhatsApp and other applications could not patch the underlying vulnerability. Because it could often take months for people to update to the latest version of their software, we set out to find solutions that would keep WhatsApp users safe, even in the event of an operating system vulnerability. At that time, we realized that a cross-platform C++ library already developed by WhatsApp to send and consistently format MP4 files (called “wamedia”) could be modified to detect files which do not adhere to the MP4 standard and might trigger bugs in a vulnerable OS library on the receiver side – hence putting a target’s security at risk. We rolled out this check and were able to protect WhatsApp users from the Stagefright vulnerability much more rapidly than by depending on users to update the OS itself. But because media checks run automatically on download and process untrusted inputs, we identified early on that wamedia was a prime candidate for using a memory safe language. <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23552" src="https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg" alt="" width="2100" height="1250" srcset="https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg 2100w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=916,545 916w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=768,457 768w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=1024,610 1024w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=1536,914 1536w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=2048,1219 2048w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=96,57 96w, https://engineering.fb.com/wp-content/uploads/2026/01/Rust-at-scale_inline.jpg?resize=192,114 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Our Solution: Rust at Scale Rather than an incremental rewrite, we developed the Rust version of wamedia in parallel with the original C++ version. We used differential fuzzing and extensive integration and unit tests to ensure compatibility between the two implementations. Two major hurdles were the initial binary size increase due to bringing in the Rust standard library and the build system support required for the diverse platforms supported by WhatsApp. WhatsApp made a long-term bet to build that support. In the end, we replaced 160,000 lines of C++ (excluding tests) with 90,000 lines of Rust (including tests). The Rust version showed performance and runtime memory usage advantages over the C++. Given this success, Rust was fully rolled out to all WhatsApp users and many platforms: Android, iOS, Mac, Web, Wearables, and more. With this positive evidence in hand, memory safe languages will play an ever increasing part in WhatsApp’s overall approach to application and user security. Over time, we’ve added more checks for non-conformant structures within certain file types to help protect downstream libraries from parser differential exploit attempts. Additionally, we check higher risk file types, even if structurally conformant, for risk indicators. For instance, PDFs are often a vehicle for malware, and more specifically, the presence of embedded files and scripting elements within a PDF further raise risks. We also detect when one file type masquerades as another, through a spoofed extension or MIME type. Finally, we uniformly flag known dangerous file types, such as executables or applications, for special handling in the application UX. Altogether, we call this ensemble of checks “Kaleidoscope.” This system protects people on WhatsApp from potentially malicious unofficial clients and attachments. Although format checks will not stop every attack, this layer of defense helps mitigate many of them. Each month, these libraries are distributed to billions of phones, laptops, desktops, watches, and browsers running on multiple operating systems for people on WhatsApp, Messenger, and Instagram. This is the largest ever deployment of Rust code to a diverse set of end-user platforms and products that we are aware of. Our experience speaks to the production-readiness and unique value proposition of Rust on the client-side. How Rust Fits In To WhatsApp’s Approach to App Security This is just one example of WhatsApp’s many investments in security. It’s why we built default end-to-end encryption for personal messages and calls, offer end-to-end encrypted backups, and use key transparency technology to verify a secure connection, provide additional calling protections, and more. WhatsApp has a strong track record of being loud when we find issues and working to hold bad actors accountable. For example, WhatsApp reports CVEs for important issues we find in our applications, even if we do not find evidence of exploitation. We do this to give people on WhatsApp the best chance of protecting themselves by seeing a security advisory and updating quickly. To ensure application security, we first must identify and quantify the sources of risk. We do this through internal and external audits like NCC Group’s public assessment of WhatsApp’s end-to-end encrypted backups, fuzzing, static analysis, supply chain management, and automated attack surface analysis. We also recently expanded our Bug Bounty program to introduce the WhatsApp Research Proxy – a tool that makes research into WhatsApp’s network protocol more effective. Next, we reduce the identified risk. Like many others in the industry, we found that the majority of the high severity vulnerabilities we published were due to memory safety issues in code written in the C and C++ programming languages. To combat this we invest in three parallel strategies: Design the product to minimize unnecessary attack surface exposure. Invest in security assurance for the remaining C and C++ code. Default the choice of memory safe languages, and not C and C++, for new code. WhatsApp has added protections like CFI, hardened memory allocators, safer buffer handling APIs, and more. C and C++ developers have specialized security training, development guidelines, and automated security analysis on their changes. We also have strict SLAs for fixing issues uncovered by the risk identification process. Accelerating Rust Adoption to Enhance Security Rust enabled WhatsApp’s security team to develop a secure, high performance, cross-platform library to ensure media shared on the platform is consistent and safe across devices. This is an important step forward in adding additional security behind the scenes for users and part of our ongoing defense-in-depth approach. Security teams at WhatsApp and Meta are highlighting opportunities for high impact adoption of Rust to interested teams, and we anticipate accelerating adoption of Rust over the coming years. The post Rust at Scale: An Added Layer of Security for WhatsApp appeared first on Engineering at Meta.
Adapting the Facebook Reels RecSys AI Model Based on User Feedback
We’ve improved personalized video recommendations on Facebook Reels by moving beyond metrics such as likes and watch time and directly leveraging user feedback. Our new User True Interest Survey (UTIS) model, now helps surface more niche, high-quality content and boosts engagement, retention, and satisfaction. We’re doubling down on personalization, tackling challenges like sparse user data [...] Read More... The post Adapting the Facebook Reels RecSys AI Model Based on User Feedback appeared first on Engineering at Meta.
We’ve improved personalized video recommendations on Facebook Reels by moving beyond metrics such as likes and watch time and directly leveraging user feedback. Our new User True Interest Survey (UTIS) model, now helps surface more niche, high-quality content and boosts engagement, retention, and satisfaction. We’re doubling down on personalization, tackling challenges like sparse user data and bias, and exploring advanced AI to make recommendations even smarter and more diverse. Our paper, “Improve the Personalization of Large-Scale Ranking Systems by Integrating User Survey Feedback” shares full details on this work. Delivering personalized video recommendations is a common challenge for user satisfaction and long-term engagement on large-scale social platforms. At Facebook Reels, we’ve been working to close this gap by focusing on “interest matching” – ensuring that the content people see truly aligns with their unique preferences. By combining large-scale user surveys with recent advances in machine learning, we are now able to better understand and model what people genuinely care about, which has led to significant improvements in both recommendation quality and overall user satisfaction. Why True Interest Matters Traditional recommendation systems often rely on engagement signals – such as likes, shares, and watch time – or heuristics to infer user interests. However, these signals can be noisy and may not fully capture the nuances of what people actually care about or want to see. Models trained only on these signals tend to recommend content that has high short-term user value measured by watch time and engagement but doesn’t capture true interests that are important for long-term utility of the product. To bridge this gap, we needed a more direct way to measure user perception of content relevance. Our research shows that effective interest matching goes beyond simple topic alignment; it also encompasses factors like audio, production style, mood, and motivation. By accurately capturing these dimensions, we can deliver recommendations that feel more relevant and personalized, encouraging people to return to the app more frequently. <img loading="lazy" decoding="async" class="wp-image-23392" src="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png" alt="" width="600" height="560" srcset="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png 1254w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png?resize=916,855 916w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png?resize=768,717 768w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png?resize=1024,955 1024w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png?resize=96,90 96w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image1.png?resize=192,179 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Recommendation systems are typically optimized based on user interactions on the product, such as watch time, likes, shares, etc. However, by incorporating user perception feedback – like interest match and novelty – we can significantly improve relevance, quality, and the overall ecosystem. How We Measured User Perception To validate our approach, we launched large-scale, randomized surveys within the video feed, asking users, “How well does this video match your interests?” These surveys were deployed across Facebook Reels and other video surfaces, enabling us to collect thousands of in-context responses from users every day. The results revealed that previous interest heuristics only achieved a 48.3% precision in identifying true interests, highlighting the need for a more robust measurement framework. By weighting responses to correct for sampling and nonresponse bias, we built a comprehensive dataset that accurately reflects real user preferences – moving beyond implicit engagement signals to leverage direct, real-time user feedback. <img loading="lazy" decoding="async" class="aligncenter wp-image-23429" src="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png" alt="" width="277" height="600" srcset="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png 924w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=423,916 423w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=768,1662 768w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=473,1024 473w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=710,1536 710w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=96,208 96w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image-updated.png?resize=192,415 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Framework: User True Interest Survey (UTIS) Model Daily, a certain proportion of users viewing sessions on the platform are randomly chosen to display a single-question survey asking, “To what extent does this video match your interests?” on a 1-5 scale. The survey aims to gather real-time feedback from users about the content they have just viewed. The main candidate ranking model used by the platform is a large multi-task, multi-label model. We trained a lightweight UTIS alignment model layer on the collected user survey responses using existing predictions of the main model as input features. The survey responses used to train our model were binarized for easy modelling and denoises variance in responses. In addition, new features were engineered to capture user behavior, content attributes, and interest signals with the object function to optimize predicting users’ interest-matching extent. The UTIS model outputs the probability that a user is satisfied with a video, and is designed to be interpretable, allowing us to understand the factors contributing to users’ interest matching experience. <img loading="lazy" decoding="async" class="wp-image-23393" src="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png" alt="" width="600" height="602" srcset="https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png 1234w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png?resize=913,916 913w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png?resize=768,770 768w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png?resize=1021,1024 1021w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png?resize=96,96 96w, https://engineering.fb.com/wp-content/uploads/2025/11/Meta-Facebook-Reels-Adapting-Recsys-image2.png?resize=192,193 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> User perception feedback collected using surveys are extremely sparse but such feedback can be generalized in large scale recommendation systems using our novel model “Perception Layer” architecture that uses existing event predictions as additional features. Integrating the UTIS Model in the Main Ranking System We have experimented with and deployed several use cases of the UTIS model in our ranking funnel, all of which showed successful tier 0 user retention metric improvements: Late Stage Ranking (LSR): UTIS is deployed in parallel to the LSR model, providing an additional input feature into the final value formula. This allows fine-tuning of the final ranking stage to incorporate true interests while balancing other concerns. Early Stage Ranking (Retrieval): UTIS is used to reconstruct users’ true interest profiles by aggregating survey data to predict affinity for any given user-video pair, allowing us to re-rank the user interest profile and source more candidates relevant to users’ true interests. Also, large sequences based on user-to-item retrieval models are aligned using knowledge distillation based objectives trained on UTIS predictions from LSR as labels. The UTIS model score is now one of the inputs to our ranking system. Videos predicted to be of high interest receive a modest boost, while those with low predicted interest are demoted. This approach has led to: Increased delivery of high-quality, niche content. A reduction in low-quality, generic popularity based recommendations. Improvements in like, share, and follow rates. Improved user engagement and retention metrics. Since launching this approach, we’ve observed robust offline and online performance Offline Performance: The UTIS model delivered an improvement in accuracy and reliability over the heuristic rule baseline. Accuracy increased from 59.5% to 71.5%, precision improved from 48.3% to 63.2%, and recall increased from 45.4% to 66.1%. These gains demonstrate the model’s ability to help in accurately identifying users’ interest preferences. Online Performance: Large-scale A/B testing with over 10 million users confirmed these improvements in real-world settings. The UTIS model consistently outperformed the baseline, driving higher user engagement and retention. Notably, we saw a +5.4% increase in high survey ratings, a -6.84% reduction in low survey ratings, a +5.2% boost in total user engagement, and a -0.34% decrease in integrity violations. These results highlight the model’s effectiveness in improving user experience and matching users with relevant interests. Future Work for Interest Recommendations By integrating survey-based measurement with machine learning, we are creating a more engaging and personalized experience – delivering content on Facebook Reels that feels truly tailored to each user and encourages repeat visits. While survey-driven modeling has already improved our recommendations, there remain important opportunities for improvement, such as better serving users with sparse engagement histories, reducing bias in survey sampling and delivery, further personalizing recommendations for diverse user cohorts and improving the diversity of recommendations. To address these challenges and continue advancing relevance and quality, we are also exploring advanced modeling techniques, including large language models and more granular user representations. Read the Paper Improve the Personalization of Large-Scale Ranking Systems by Integrating User Survey Feedback .meta-btn { background-color: #0064E0; /* Meta Blue */ color: #ffffff !important; /* Force white text */ padding: 10px 20px; /* Button size */ border: none; /* No border */ border-radius: 5px; /* Rounded corners */ cursor: pointer; /* Pointer cursor on hover */ text-decoration: none; /* Remove underline */ display: inline-block; /* Button-like display */ font-weight: bold; /* Optional: bold text */ transition: color 0.3s ease, background-color 0.3s ease; /* Smooth transitions */ } .meta-btn:hover { color: #808080 !important; /* Grey text on hover */ } The post Adapting the Facebook Reels RecSys AI Model Based on User Feedback appeared first on Engineering at Meta.
Build a large enough website with a large enough codebase, and you’ll eventually find that CSS presents challenges at scale. It’s no different at Meta, which is why we open-sourced StyleX, a solution for CSS at scale. StyleX combines the ergonomics of CSS-in-JS with the performance of static CSS. It allows atomic styling of components [...] Read More... The post CSS at Scale With StyleX appeared first on Engineering at Meta.
Build a large enough website with a large enough codebase, and you’ll eventually find that CSS presents challenges at scale. It’s no different at Meta, which is why we open-sourced StyleX, a solution for CSS at scale. StyleX combines the ergonomics of CSS-in-JS with the performance of static CSS. It allows atomic styling of components while deduplicating definitions to reduce bundle size and exposes a simple API for developers. StyleX has become the standard at companies like Figma and Snowflake. Here at Meta, it’s the standard styling system across Facebook, Instagram, WhatsApp, Messenger, and Threads. On this episode of the Meta Tech Podcast, meet Melissa, a software engineer at Meta and one of StyleX’s maintainers. Pascal Hartig talks to her about all things StyleX—its origins, how open source has been a force multiplier for the project, and what it’s like interacting with large companies across the industry as they’ve adopted StyleX. Download or listen to the episode below: You can also find the episode wherever you get your podcasts, including: Spotify Apple Podcasts Pocket Casts The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features. Send us feedback on Instagram, Threads, or X. And if you’re interested in learning more about career opportunities at Meta visit the Meta Careers page. The post CSS at Scale With StyleX appeared first on Engineering at Meta.
Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption
The 2025 Typed Python Survey, conducted by contributors from JetBrains, Meta, and the broader Python typing community, offers a comprehensive look at the current state of Python’s type system and developer tooling. With 1,241 responses (a 15% increase from last year), the survey captures the evolving sentiment, challenges, and opportunities around Python typing in the [...] Read More... The post Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption appeared first on Engineering at Meta.
The 2025 Typed Python Survey, conducted by contributors from JetBrains, Meta, and the broader Python typing community, offers a comprehensive look at the current state of Python’s type system and developer tooling. With 1,241 responses (a 15% increase from last year), the survey captures the evolving sentiment, challenges, and opportunities around Python typing in the open-source ecosystem. In this blog we’ll cover a summary of the key findings and trends from this year’s results. Who Responded? The survey was initially distributed on official social media accounts by the survey creators, and subsequently shared organically across further platforms including Reddit, email newsletters, Mastodon, LinkedIn, Discord, and Twitter. When respondents were asked which platform they heard about the survey from, Reddit emerged as the most effective channel, but significant engagement also came from email newsletters and Mastodon, reflecting the diverse spaces where Python developers connect and share knowledge. The respondent pool was predominantly composed of developers experienced with Python and typing. Nearly half reported over a decade of Python experience, and another third had between five and 10 years. While there was representation from newcomers, the majority of participants brought substantial expertise to their responses. Experience with type hints was similarly robust, with most respondents having used them for several years and only a small minority indicating no experience with typing. Typing Adoption and Attitudes The survey results reveal that Python’s type hinting system has become a core part of development for most engineers. An impressive 86% of respondents report that they “always” or “often” use type hints in their Python code, a figure that remains consistent with last year’s Typed Python survey. For the first time this year the survey also asked participants to indicate how many years of experience they have with Python and with Python typing. We found that adoption of typing is similar across all experience levels, but there are some interesting nuances: Developers with 5–10 years of Python experience are the most enthusiastic adopters, with 93% reporting regularly using type hints. Among the most junior developers (0–2 years of experience), adoption is slightly lower at 83%. Possible reasons for this could be the learning curve for newcomers (repeatedly mentioned in later survey questions). For senior developers (10+ years of experience), adoption was the lowest of all cohorts, with only 80% reporting using them always or often. Reasons for this drop are unclear, it could reflect more experienced python developers having gotten used to writing Python without type hints before they were supported, or possibly they are more likely to work on larger or legacy codebases that are difficult to migrate. <img loading="lazy" decoding="async" class="size-full wp-image-23493" src="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png" alt="" width="2428" height="936" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png 2428w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=916,353 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=768,296 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=1024,395 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=1536,592 1536w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=2048,790 2048w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=96,37 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-1.png?resize=192,74 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Percent of respondents who use types “often” or “always,” segmented by years of Python experience. Overall, the data shows that type hints are widely embraced by the Python community, with strong support from engineers at all experience levels. However, we should note there may be some selection bias at play here, as it’s possible developers who are more familiar with types and use them more often are also more likely to be interested in taking a survey about it. Why Developers Love Python Typing When asked what developers loved about the Python type system there were some mixed reactions, with a number of responses just stating, “nothing” (note this was an optional question). This indicates the presence of some strong negative opinions towards the type system among a minority of Python users. The majority of responses were positive, with the following themes emerging prominently: Optionality and Gradual Adoption: The optional nature of the type system and the ability to adopt it incrementally into existing projects are highly valued, allowing flexibility in development. Improved Readability and Documentation: Type hints serve as in-code documentation, making code clearer and easier to read, understand, and reason about for both the author and other developers, especially in larger codebases. Enhanced Tooling and IDE Support: The type system significantly improves IDE features like autocomplete/IntelliSense, jump-to-definition, and inline type hints, leading to a better developer experience. Bug Prevention and Code Correctness: It helps catch errors and subtle bugs earlier during development or refactoring, increasing confidence and leading to more robust and reliable code. Flexibility and Features: Respondents appreciate the flexibility, expressiveness, and powerful features of the system, including protocols, generics (especially the new syntax), and the ability to inspect annotations at runtime for use with libraries like Pydantic/FastAPI. <img loading="lazy" decoding="async" class="size-full wp-image-23488" src="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png" alt="" width="1280" height="720" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png 1280w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=916,515 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-2.png?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Sample of responses to the question, “What do you love about Python Typing?” Challenges and Pain Points In addition to assessing positive sentiment towards Python typing, we also asked respondents what challenges and pain points they face. With over 800 responses to the question, “What is the hardest part about using the Python type system?” the following themes were identified: Third-Party Library/Framework Support: Many respondents cited the difficulty of integrating types with untyped, incomplete, or incorrect type annotations in third-party libraries (e.g., NumPy, Pandas, Django). Complexity of Advanced Features: Advanced concepts such as generics, TypeVar (including co/contravariance), callables/decorators, and complex/nested types were frequently mentioned as difficult to understand or express. Tooling and Ecosystem Fragmentation: The ecosystem is seen as chaotic, with inconsistencies between different type checkers (like Mypy and Pyright), slow performance of tools like Mypy, and a desire for an official, built-in type checker. Lack of Enforcement and Runtime Guarantees: The fact that typing is optional and is not enforced at runtime or by the Python interpreter makes it harder to convince others to use it, enforce its consistent use, and fully trust the type hints. Verbosity and Code Readability: The necessary type hints, especially for complex structures, can be verbose, make the code less readable, and feel non-Pythonic. Dealing with Legacy/Dynamic Code: It is hard to integrate typing into old, untyped codebases, particularly when they use dynamic Python features that do not play well with static typing. Type System Limitations and Evolution: The type system is perceived as incomplete or less expressive than languages like TypeScript, and its rapid evolution means syntax and best practices are constantly changing. Most Requested Features A little less than half of respondents had suggestions for what they thought was missing from the Python type system, the most commonly requested features being: Missing Features From TypeScript and Other Languages: Many respondents requested features inspired by TypeScript, such as Intersection types (like the & operator), Mapped and Conditional types, Utility types (like Pick, Omit, keyof, and typeof), and better Structural typing for dictionaries/dicts (e.g., more flexible TypedDict or anonymous types). Runtime Type Enforcement and Performance: A significant number of developers desire optional runtime type enforcement or guarantees, as well as performance optimizations (JIT/AOT compilation) based on the type hints provided. Better Generics and Algebraic Data Types (ADTs): Requests include features like higher-kinded types (HKT), improved support for TypeVarTuple (e.g., bounds and unpacking), better generics implementation, and official support for algebraic data types (e.g., Result, Option, or Rust-like enums/sum types). Improved Tooling, Consistency, and Syntax: Developers asked for an official/built-in type checker that is fast and consistent, a less verbose syntax for common patterns like nullable types (? instead of | None) and callables, and better support/documentation for complex types (like nested dicts, NumPy/Pandas arrays). Handling of Complex/Dynamic Patterns: Specific missing capabilities include better support for typing function wrappers/decorators (e.g., using ParamSpec effectively), being able to type dynamic attributes (like those added by Django/ORMs), and improved type narrowing and control flow analysis. Tooling Trends The developer tooling landscape for Python typing continues to evolve, with both established and emerging tools shaping how engineers work. Mypy remains the most widely used type checker, with 58% of respondents reporting using it. While this represents a slight dip from 61% in last year’s survey, Mypy still holds a dominant position in the ecosystem. At the same time, new Rust-based type checkers like Pyrefly, Ty, and Zuban are quickly gaining traction, now used by over 20% of survey participants collectively. <img loading="lazy" decoding="async" class="size-full wp-image-23487" src="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png" alt="" width="1999" height="1125" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png 1999w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=916,516 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-3.png?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> The top six most popular answers to the question, “What type checking tools do your projects use (select all that apply)?” When it comes to development environments, VS Code leads the pack as the most popular IDE among Python developers, followed by PyCharm and (Neo)vim/vim. The use of type checking tools within IDEs also mimics the popularity of the IDE themselves, with VS Code’s default (Pylance/Pyright) and PyCharm’s built-in support being the first and third most popular options respectively. How Developers Learn and Get Help When it comes to learning about Python typing and getting help, developers rely on a mix of official resources, community-driven content, and AI-powered tools, a similar learning landscape to what we saw in last year’s survey. <img loading="lazy" decoding="async" class="size-full wp-image-23490" src="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png" alt="" width="1999" height="1125" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png 1999w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=916,516 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Python-typing-survey-2025-image-4.png?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> Top six responses to the question, “How do you learn Python typing (select all that apply)?” Official documentation remains the go-to resource for most developers. The majority of respondents reported learning about Python typing through the official docs, with 865 citing it as their primary source for learning and 891 turning to it for help. Python’s dedicated typing documentation and type checker-specific docs are also heavily used, showing that well-maintained, authoritative resources are still highly valued. Blog posts have climbed in popularity, now ranking as the second most common way developers learn about typing, up from third place last year. Online tutorials, code reviews, and YouTube videos also play a significant role. Community platforms are gaining traction as sources for updates and new features. Reddit, in particular, has become a key channel for discovering new developments in the type system, jumping from fifth to third place as a source for news. Email newsletters, podcasts, and Mastodon are also on the rise. Large language models (LLMs) are now a notable part of the help-seeking landscape. Over 400 respondents reported using LLM chat tools, and nearly 300 use in-editor LLM suggestions when working with Python typing. Opportunities and Next Steps The 2025 Python Typing Survey highlights the Python community’s sustained adoption of typing features and tools to support their usage. It also points to clear opportunities for continued growth and improvement, including: Increasing library coverage: One of the most consistent requests from the community is for broader and deeper type annotation coverage in popular libraries. Expanding type hints across widely used packages will make static typing more practical and valuable for everyone. Improving documentation: While official documentation remains the top resource, there’s a strong appetite for more discoverable and accessible learning materials. Leveraging channels like newsletters, blog posts, and Reddit can help surface new features, best practices, and real-world examples to a wider audience. Clarify tooling differences: The growing variety of type checkers and tools is a sign of a healthy ecosystem, but can also reflect a lack of consensus/standardisation and can be confusing for users. There’s an opportunity to drive more consistency between tools or provide clearer guidance on their differences and best-fit use cases. To learn more about Meta Open Source, visit our website, subscribe to our YouTube channel, or follow us on Facebook, Threads, X, Bluesky and LinkedIn. Acknowledgements This survey ran from 29th Aug to 16th Sept 2025 and received 1,241 responses in total. Thanks to everyone who participated! The Python typing ecosystem continues to evolve, and your feedback helps shape its future. Also, special thanks to the Jetbrains PyCharm team for providing the graphics used in this piece. The post Python Typing Survey 2025: Code Quality and Flexibility As Top Reasons for Typing Adoption appeared first on Engineering at Meta.
DrP: Meta’s Root Cause Analysis Platform at Scale
Incident investigation can be a daunting task in today’s digital landscape, where large-scale systems comprise numerous interconnected components and dependencies DrP is a root cause analysis (RCA) platform, designed by Meta, to programmatically automate the investigation process, significantly reducing the mean time to resolve (MTTR) for incidents and alleviating on-call toil Today, DrP is used [...] Read More... The post DrP: Meta’s Root Cause Analysis Platform at Scale appeared first on Engineering at Meta.
Incident investigation can be a daunting task in today’s digital landscape, where large-scale systems comprise numerous interconnected components and dependencies DrP is a root cause analysis (RCA) platform, designed by Meta, to programmatically automate the investigation process, significantly reducing the mean time to resolve (MTTR) for incidents and alleviating on-call toil Today, DrP is used by over 300 teams at Meta, running 50,000 analyses daily, and has been effective in reducing MTTR by 20-80% By understanding DrP and its capabilities, we can unlock new possibilities for efficient incident resolution and improved system reliability. What It Is DrP is an end-to-end platform that automates the investigation process for large-scale systems. It addresses the inefficiencies of manual investigations, which often rely on outdated playbooks and ad-hoc scripts. These traditional methods can lead to prolonged downtimes and increased on-call toil as engineers spend countless hours triaging and debugging incidents. DrP offers a comprehensive solution by providing an expressive and flexible SDK to author investigation playbooks, known as analyzers. These analyzers are executed by a scalable backend system, which integrates seamlessly with mainstream workflows such as alerts and incident management tools. Additionally, DrP includes a post-processing system to automate actions based on investigation results, such as mitigation steps. <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23498" src="https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png" alt="" width="1999" height="1125" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png 1999w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=916,516 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-1.png?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> DrP’s key components include: Expressive SDK: The DrP SDK allows engineers to codify investigation workflows into analyzers. It provides a rich set of helper libraries and machine learning (ML) algorithms for data access and problem isolation analysis, such as anomaly detection, event isolation, time series correlation and dimension analysis. Scalable backend: The backend system executes the analyzers, providing both multi-tenant and isolated execution environments. It ensures that analyzers can be run at scale, handling thousands of automated analyses per day. Integration with workflows: DrP integrates with alerting and incident management tools, allowing for the auto-triggering of analyzers on incidents. This integration ensures that investigation results are immediately available to on-call engineers. Post-processing system: After an investigation, the post-processing system can take automated actions based on the analysis results. For example, it can create tasks or pull requests to mitigate issues identified during the investigation. How It Works Authoring Workflow <img loading="lazy" decoding="async" class="alignnone size-full wp-image-23499" src="https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg" alt="" width="1999" height="1125" srcset="https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg 1999w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=580,326 580w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=916,516 916w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=768,432 768w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=1024,576 1024w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=1536,864 1536w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=96,54 96w, https://engineering.fb.com/wp-content/uploads/2025/12/Meta-DrP-image-2.jpg?resize=192,108 192w" sizes="auto, (max-width: 992px) 100vw, 62vw" /> The process of creating automated playbooks, or analyzers, begins with the DrP SDK. Engineers enumerate the investigation steps, listing inputs and potential paths to isolate problem areas. The SDK provides APIs and libraries to codify these workflows, allowing engineers to capture all required input parameters and context in a type-safe manner. Enumerate investigation steps: Engineers start by listing the steps required to investigate an incident, including inputs and potential paths to isolate the problem. Bootstrap code: The DrP SDK provides bootstrap code to create a template analyzer with pre-populated boilerplate code. Engineers extend this code to capture all necessary input parameters and context. Data access and analysis: The SDK includes libraries for data access and analysis, such as dimension analysis and time series correlation. Engineers use these libraries to code the main investigation decision tree into the analyzer. Analyzer chaining: For dependent service analysis, the SDK’s APIs allow for seamless chaining of analyzers, passing context and obtaining outputs. Output and post-processing: The output method captures findings from the analysis, using special data structures for both text and machine-readable formats. Post-processing methods automate actions based on analyzer findings. Once created, analyzers are tested and sent for code review. DrP offers automated backtesting integrated into code review tools, ensuring high-quality analyzers before deployment. Consumption Workflow In production, analyzers integrate with tools like UI, CLI, alerts, and incident management systems. Analyzers can automatically trigger upon alert activation, providing immediate results to on-call engineers and improving response times. The DrP backend manages a queue for requests and a worker pool for secure execution, with results returning asynchronously. Integration with alerts: DrP is integrated with alerting systems, allowing analyzers to trigger automatically when an alert is activated. This provides immediate analysis results to on-call engineers. Execution and monitoring: The backend system manages a queue for analyzer requests and a worker pool for execution. It monitors execution, ensuring that analyzers run securely and efficiently. Post-processing and insights: A separate post-processing system handles analysis results, annotating alerts with findings. The DrP Insights system periodically analyzes outputs to identify and rank top alert causes, aiding teams in prioritizing reliability improvements. Why It Matters Reducing MTTR DrP has demonstrated significant improvements in reducing MTTR across various teams and use cases. By automating manual investigations, DrP enables faster triage and mitigation of incidents, leading to quicker system recovery and improved availability. Efficiency: Automated investigations reduce the time engineers spend on manual triage, allowing them to focus on more complex tasks. This efficiency translates to faster incident resolution and reduced downtime. Consistency: By codifying investigation workflows into analyzers, DrP ensures consistent and repeatable investigations. This consistency reduces the likelihood of errors and improves the reliability of incident resolution. Scalability: DrP can handle thousands of automated analyses per day, making it suitable for large-scale systems with complex dependencies. Its scalability ensures that it can support the needs of growing organizations. Enhancing On-Call Productivity The automation provided by DrP reduces the on-call effort during investigations, saving engineering hours and reducing on-call fatigue. By automating repetitive and time-consuming steps, DrP allows engineers to focus on more complex tasks, improving overall productivity. Scalability and Adoption DrP has been successfully deployed at scale at Meta, covering over 300 teams and 2000 analyzers, executing 50,000 automated analyses per day. Its integration into mainstream workflows, such as alerting systems, has facilitated widespread adoption and demonstrated its value in real-world scenarios. Widespread adoption: DrP has been adopted by hundreds of teams across various domains, demonstrating its versatility and effectiveness in addressing diverse investigation needs. Proven impact: DrP has been in production for over five years, with proven results in reducing MTTR and improving on-call productivity. Its impact is evident in the positive feedback received from users and the significant improvements in incident resolution times. Continuous improvement: DrP is continuously evolving, with ongoing enhancements to its ML algorithms, SDK, backend system, and integrations. This commitment to continuous improvement ensures that DrP remains a cutting-edge solution for incident investigations, while its growing adoption across teams enables existing workflows and analyzers to be reused by others, compounding the shared knowledge base and making it increasingly valuable across the organization. What’s Next Looking ahead, DrP aims to evolve into an AI-native platform, playing a central role in advancing Meta’s broader AI4Ops vision, enabling more powerful and automated investigations. This transformation will enhance analysis by delivering more accurate and insightful results, while also simplifying the user experience through streamlined ML algorithms, SDKs, UI, and integrations facilitating effortless authoring and execution of analyzers. Read the Paper DrP: Meta’s Efficient Investigations Platform at Scale Acknowledgements We wish to thank contributors to this effort across many teams throughout Meta Team – Eduardo Hernandez, Jimmy Wang, Akash Jothi, Kshitiz Bhattarai, Shreya Shah, Neeru Sharma, Alex He, Juan-Pablo E, Oswaldo R, Vamsi Kunchaparthi, Daniel An, Rakesh Vanga, Ankit Agarwal, Narayanan Sankaran, Vlad Tsvang, Khushbu Thakur, Srikanth Kamath, Chris Davis, Rohit JV, Ohad Yahalom, Bao Nguyen, Viraaj Navelkar, Arturo Lira, Nikolay Laptev, Sean Lee, Yulin Chen Leadership – Sanjay Sundarajan, John Ehrhardt, Ruben Badaro, Nitin Gupta, Victoria Dudin, Benjamin Renard, Gautam Shanbhag, Barak Yagour, Aparna Ramani The post DrP: Meta’s Root Cause Analysis Platform at Scale appeared first on Engineering at Meta.