Bug hunting is hard and can sometimes go unnoticed across our industry. Building scalable bug detection methods across large codebases and open source libraries is an underappreciated yet critical effort every engineering company has to work through. Because the ideal outcome is that bugs are found and fixed before they are exploited, some of our industry’s best work is often unknown.
However, information sharing about bug discoveries between our engineers, or between bug bounty researchers, has generated new ideas and led to novel ways to improve the security of our platform. We believe that sharing our key learnings externally on a more regular basis is an important step forward for transparency and advancement across our industry.
Today, we’re excited to publish the first in a series of regular Bug Bulletins, in which we’ll share the details of some of the notable finds identified in our own, and in third-party, code. We’ll use these bulletins to share overviews — and occasionally deep dives — into specific bugs, as well as updates on relevant security programs.
How we hunt for bugs at Meta
Before we dive into recent learnings, here is a bit of context on how we look for security bugs at Meta: Finding and fixing bugs is an essential part of our “defense-in-depth” security strategy. This approach means we’ve implemented layers of protections across our platform. If a vulnerability in our code makes it past one line of defense, we have additional layers of support to prevent and address bugs as quickly as possible.
As it’s practically impossible to write flawless code, it’s not uncommon for software — both in-house and open source — to have bugs. Like our peers, our strategy is twofold: While we’re constantly working on improving the quality of our code before it makes it into production, we also build in layers of protection to prevent and detect bugs in existing code as quickly as possible. We rely on a combination of automated tooling, security reviews, red teaming, and our Bug Bounty program to help keep our community safe. This work isn’t done in a vacuum or within one single team. And often, bugs that are discovered in one product area can inform the way we approach our work in an entirely different part of the codebase.
In this first post in the series, we’ll share notable bugs that we’ve found and fixed in our own software and reported to third-party developers, and we’ll highlight some of the finds by our bug bounty researchers.
Static analysis work
Our automated static analysis tools help us review large amounts of code at scale, which frees up time for our engineers to analyze more complex scenarios. Over the past five years, we’ve built and continuously improved our homegrown tools — they now help us find about 70 percent of the security vulnerabilities we discover. For example, our work on Zoncolan, which reviews Hack code, has helped us find over 1,300 bugs this year.
Zoncolan recently identified a bug in Messenger’s back-end production that could have allowed a bad actor to inject “You celebrated a friendversary with a friend” into a chat thread between any two people. This could have been leveraged by a malicious actor to create a false sense of long-term authentic connection to exploit for scams. The message would have appeared inside someone’s Chat List if they were already connected or in their Message Request folder if they weren’t. Zoncolan found this bug by observing that our systems were accepting user input and creating a viewer context out of it. These vulnerable endpoints did not have any front-end integration and were not present anywhere in our apps in production. Our team quickly reviewed the report from Zoncolan, found no evidence of abuse, and mitigated the impacted endpoints to fix the issue.
Red Team X work
In addition to bug hunting in our own code, our Red Team X works to spot security vulnerabilities in external hardware and software and keep the broader internet safe. As part of our responsible disclosure policy, we regularly report bugs in third-party code to companies and work directly with them to test and confirm their mitigations. Here are a few recent examples of interesting collaborations we’ve had across the industry.
Our team is working with Schneider Electric, a company that specializes in digital automation and energy management, to fix a set of vulnerabilities in two Ethernet modules that bring modern networking capabilities to the M580 line of PLCs (programmable logic controllers). PLCs or PACs (programmable automation controllers) allow equipment to act on complex instructions and are used in industrial control systems for machinery across many industries. When chained together, the identified vulnerabilities could enable an authenticated attacker to bypass multiple firmware verification stages and the module’s secure boot process, remotely overwriting the module’s firmware. This could lead to either permanently bricking or backdooring the device. To exploit these vulnerabilities, an attacker would require network access and valid credentials for the privileged “installer” account. Schneider Electric is performing a comprehensive analysis to identify and provide fixes for all products affected by these findings. These vulnerabilities have been assigned CVE-2022-34759, CVE-2022-34760, CVE-2022-34761, CVE-2022-34762, CVE-2022-34763, CVE-2022-34764, and CVE-2022-34765.
We also worked with Airspan, a company that provides hardware and software for 4G and 5G networks, to harden a line of eNodeBs, the wireless access points that provide cell service to phones and other LTE devices and allow them to connect to the network. We reported issues and verified fixes for vulnerabilities that could have allowed both local and networked adversaries to gain root command execution. By controlling the “last hop” of LTE infrastructure, an attacker could have impacted availability by disabling cell service. Our investigation found that while an attacker could have accessed the encrypted user traffic, they would not have been able to decrypt it. A malicious actor could have used their access to pivot further into cellular infrastructure to further impact the company networks where these eNodeBs are deployed, or even to connect to major network operators that route and handle LTE data and calls. These issues have been assigned CVE-2022-36306, CVE-2022-36307, CVE-2022-36308, CVE-2022-36309, CVE-2022-36310, CVE-2022-36311, and CVE-2022-36312.
Our Red Team X recently reported a vulnerability in Apple’s Big Sur operating system that allows for kernel code execution in Darwin kernels. This could have been exploited by forking processes and leveraging the host_request_notification API. We found this bug through a manual audit of Darwin Kernel and experimentation. Apple disclosed the vulnerability and released software fixes in Security Update 2021-005 Catalina; iOS 14.8 and iPadOS 14.8; tvOS 15; iOS 15 and iPadOS 15; watchOS 8; and macOS Big Sur 11.6. This finding was assigned CVE-2021-30857.
Through manual code review, Red Team X also found a series of bugs in EternalTerminal, a third-party open source remote terminal service that can automatically reconnect after events like network outages and IP roaming without interrupting the session. Most notably, they reported a bug that could allow the attacker to change ownership permissions on arbitrary files and subsequently gain root privileges on the host machine. These bugs have been patched by the maintainers of EternalTerminal and assigned CVE-2022-24949, CVE-2022-24950, CVE-2022-24951, and CVE-2022-24952.
Bug bounty work
One benefit of having a 10-plus-year Bug Bounty program is that some of our researchers have dedicated years to hunting on our platform and have become extremely familiar with our products and services. These researchers are able to dig beyond surface-level issues and help us identify impactful but niche bugs that the broader community wouldn’t necessarily know to look for.
For example, one of our longtime researchers, Youssef Sammouda, spent seven months hunting for bugs on our Facebook API, known as Canvas App. Over a decade ago, we created the Canvas App to help developers embed their games and apps on our web gaming platform. Sammouda audited the client-side code inside apps.facebook.com and found a number of high-severity bugs in our OAuth implementation that could have led to an account takeover scenario. These were complex scenarios where, to attempt to exploit the bugs, an attacker would need to trick their target into clicking a link, for example. We investigated these reports, fixed the issues, and found no evidence of abuse. In total, Sammouda submitted six bug reports to us, totaling $187,250 in bounty awards. To date, this is the largest series of payouts we’ve issued to one researcher for a single implementation. We’ve worked with him to confirm our mitigations, and we appreciate his continued partnership in testing our defenses. Sammouda shared additional insights and advice for security researchers in his blog.
Another of our longtime researchers, Philippe Harewood, recently identified an endpoint vulnerability that could have allowed a malicious actor to retrieve an Instagram app access token. The great benefit of working through these reports with our researchers is that even when we have protections in place to prevent the most impactful exploit scenarios — like in this case — collaboration often leads us to make changes to prevent similar issues in the future across our codebase, and we reward researchers on the potential maximum impact we find as a result of our own internal security research. We fixed this endpoint issue, have not seen any indicators of abuse, and awarded Harewood a $30,000 bounty award. Harewood shared additional insights on his blog.
We are thankful to the security community for contributing great research to our program. We’re continuing to do our part to expand the pool of researchers to join the bug bounty community. We recently hosted our inaugural BountyConEDU conference in Madrid, where we invited university students and recent graduates across Europe to learn how we investigate reports and to participate in a live hacking event. Over three days, we received 26 valid reports and paid out more than $35,000 in bounties. This pilot event far surpassed our expectations, and we are excited to use our learnings to create similar opportunities around the world.
As we continue to shape the format of these Bug Bulletin updates, we welcome any feedback from our industry partners and peers and hope they will let us know what they’d find most beneficial to read about next on the Engineering at Meta blog.
