For years, adversaries took advantage of organizations’ reluctance to talk about defensive security techniques, meaning everyone was trying to solve the same problems on their own. But now the security industry is moving toward more openness and collaboration in threat intelligence, problem-solving, tooling, and other areas. We’re all better off because we learn from one another.
To facilitate more collaboration, Facebook’s Security @Scale events are designed to bring different corners of the engineering and security communities together to share the latest research, technical developments, and stories from the front lines of security.
The fourth security-focused @Scale conference took place this week in Boston, home to an incredible engineering community that is developing innovations for security. Talks this year focused on engineering security and highlighted the importance of combining multiple engineering disciplines to tackle complex security challenges. The speakers shared their unique perspectives on security and represented a diversity of backgrounds, including data science, electrical engineering, software engineering, and computer forensics.
Check out selected videos and presentation summaries from the event below, featuring experts from Akamai, Bit 9 + Carbon Black, Brightcove, Facebook, HubSpot, Lookout, and Security Innovation.
Engineering Security at Facebook
Facebook engineering director Gregg Stefancik leads the Security Infrastructure team responsible for making it easier for engineers to choose the safe and secure option so that not all of them have to be security experts. Gregg highlighted how his team uses automation and tooling to create frameworks for secure development and threat detection.
About five years ago, a group of engineers at Facebook took it upon themselves to move the state of our frameworks and APIs forward. They had a personal interest in security and started refactoring code to make it safer by default. This was the beginning of the Security Infrastructure team at Facebook.
“We want to move fast, but we can’t compromise security,” said Gregg. “So we created a team of builders, strong engineers already at Facebook, to build tools that make it very hard or impossible for developers to introduce vulnerabilities into our code.”
The new team also enabled Facebook to readily bring together other disciplines that weren’t traditional security roles, such as program analysts, through our bootcamp process for new hires.
Making Security Usable at HubSpot
Senior software engineer Ken Breeman told the audience how the age-old friction between security and usability is inspiring creative solutions to security challenges at HubSpot.
Ken focused on how HubSpot engineers have worked to improve both security and usability for authentication, including optimizing self-service options for password reset tools, using SSO strategically, and creating requirements for sufficiently robust passwords that users can remember. He also discussed how HubSpot built systems and tools so that engineers could accelerate privilege escalation and code reviews in emergency cases.
In closing, Ken challenged the audience to “find a point of friction … and find a new way to scale it, a new way to improve it.”
Safety at Scale
Kevin Riggle, a senior security researcher at Akamai, discussed what his team has learned from other disciplines — including nuclear engineering, transportation operations, and aviation — about addressing safety challenges associated with building complex systems. While software systems move faster and with greater scale than complex systems in other industries, software developers aren’t the first organizations to build large, highly interdependent systems on which both money and lives depend.
“A great body of literature says safety at scale is impossible,” Kevin said. “But safer software systems are possible when we understand and apply concepts inherently associated with complex systems, such as emergent behavior.”
Kevin also highlighted Boston’s cross-industry community, focused on building safe, complex systems, and encouraged the audience to read Engineering a Safer World by Nancy G. Leveson, professor of aeronautics and astronautics and engineering systems at MIT. The free PDF is available from The MIT Press here.
Elliptic Curve Cryptography
Deirdre Connolly is a software engineer at Brightcove, and she shared her experience contributing to a new Crypto Forum Research Group (CFRG) draft, “Elliptic Curves for Security.” Research efforts like this underpin the interoperability of our distributed systems and allow us to operate securely at scale. Security standards and research groups like CFRG are accessible to anyone who wants to be part of the security discussion.
“Everyone can contribute to the standards that drive the security and scalability for our future systems,” said Deirdre. “You can edit a document or just listen to what’s being discussed. Getting involved boosts your knowledge and capabilities as an engineer. And that’s a huge win.”
Deirdre also provided an overview of elliptic curve cryptography and why the new protocol is important to the longevity of private keys.
Building Open Source Software for Security
Facebook security engineer Javier Marcos shared technical details on how his team built osquery, a Linux and OS X intrusion detection and response tool. In his presentation, Javier discussed how the Facebook security team enabled GitHub contributors to safely submit C/C++/bash code to its continuous integration (CI) and build server. He also outlined Facebook’s CI hardening process and the attack and vulnerability reports the team receives through our bug bounty program.
Javier comes from an offensive security background, which he says made a lot of sense. “It was a logical move to go from an offensive background to a defensive role to just build things right.” He said this made it easy for Facebook to scale osquery because the system was built for security from the very beginning.
When working with open source software in security, Javier offered the following advice: Isolate and audit everything, leverage GitHub for APIs and best practices, and do not allow pull requests to build without a code review by project admins.
Rapid Identification and Classification of Mobile Malware
Seth Hardy is a staff security analyst at Lookout whose research and response team builds tools to automate the process of identifying and classifying malware for more than 60,000 new Android applications per day. His team is responsible for identifying and preventing both current and future threats to user privacy and security.
Seth described Lookout’s process for detecting, triaging, and analyzing malware ranging from commodity to highly targeted threats for more than 17 million Android applications. Lookout analyzes characteristics such as code distance of each malware sample, clusters malware groups according to relationships and code attributes, and scans for similar malware across a massive database of samples.
Seth also mentioned areas of future development, such as machine learning. “We’re trying to get more application features in for classification to determine type and family automatically,” he said. “We also want to do code distance metrics without fixed starting points, so rather than meeting those indicators, we have a system where we can take any application and look at what it’s close to and catch things that way.”
Improving Code Health with Invariant Detector
Facebook software engineer Marjori Pomarole works on Facebook’s Security Infrastructure team in London and gave an overview of Invariant Detector, a security tool she helped build to automatically enforce permission checks and protect information on Facebook from malicious actors. Invariant Detector was built to spot vulnerabilities before they’re exploited, and it adds an additional layer of automated security by augmenting code reviews and privacy frameworks to protect people who are using Facebook.
Because of Facebook’s scale, we can use dynamic code analysis on our live production traffic to automatically infer invariants to find malicious actors and block them from manipulating people’s data. Marjori demonstrated some of the ways Invariant Detector can block entire classifications of unauthorized actions on Facebook by identifying patterns in reports received through the Facebook bug bounty program.
“We push code multiple times a day, which makes it more likely that developers will introduce bugs,” she said. “Our team wants to take care of this problem for them so they can continue to move fast.”
Visualizing Security Data at Scale
John Langton, director of data science, and Alex Baker, technical director, from Bit9 + Carbon Black discussed challenges associated with visualizing large amounts of cybersecurity data, including scalability. They demonstrated how many attempts to present this data fall short and illustrated how engineers can make this data more accessible, using specific visualization techniques like tree maps and coordinated views.
Visualization enables security teams to leverage visual processing and make more informed decisions in several areas, including monitoring, hunting, and forensics. John and Alex also shared insights on data visualization methods that scale and existing problems suited for engineers to address, such as building your own tools.
“Anything you can buy off the shelf right now is going to load everything into working memory,” said John. “So if you’re really dealing with data at scale, you’re going to have to home-cook this stuff.” He advised engineers to leverage their data store for projection and modeling.
Trusted Computing
Sahil Rihan, software engineer at Facebook, outlined some of the challenges and strategies for building servers that are secure from the ground up. He highlighted the volume of potential threats present below the application layer, from compromised firmware to modified kernels and hardware implants. Because networks, software, and hardware can all be compromised, trust in the CPU is critical.
Sahil also provided an overview of testing methods available for building trust in hardware. When it comes to hardware security, Sahil said system designers must first ask themselves, “What do we trust, why do we trust it, and how does that trust begin?”
Tool or Hacker: Which One Should I Use?
Do you know how hackers find vulnerabilities in your code? Geoff Vaughan, security engineer at Security Innovation, reminded developers that it can happen anywhere throughout the software development life cycle. While some tools in the hacker arsenal are great at finding injection-based vulnerabilities, they are inadequate for detecting issues with authentication and authorization or violations of business rules — areas where hackers thrive.
Geoff shared insights for software engineers on how to think more like a hacker and effectively use a combination of pen testing and tooling at each stage of development to create more secure software.
“An informed hacker will know how to use each tool and when to rely on their hacker instincts,” Geoff said. “Learn to think like a hacker to make better tools, and approach your application as a hacker might.”
