The second annual @Scale Conference took place in San Jose this week, with 1,400 engineers coming together to share best practices for building systems and applications at scale. Engineers and technical leaders from more than 400 companies were represented among the attendees, and more than a dozen companies gave talks, including Airbnb, Box, Facebook, GitHub, Google, Instagram, LinkedIn, Microsoft, Pinterest, Pixar Animation Studios, Twitter, Uber, and WhatsApp.
Box, LinkedIn, and WhatsApp participated in a panel discussion moderated by Wired’s Cade Metz, covering lessons on how to create a flexible infrastructure, deliver the highest-quality performance, and build solutions (and teams) efficiently. During the data, mobile, and dev tools track sessions, engineers shared what they’ve learned in building in-house data systems, streaming systems, source control management models, mobile apps for unreliable networks, and much, much more.
Check out the talks below to hear from some of the industry’s leading voices on mobile, data, and dev tools engineering.
Mobile Track
Key themes during the @Scale mobile track focused on both performance and development — including fluid mobile UIs, strategies for dealing with unreliable networks, and mass messaging.
Facebook: What’s New in React Native?
Following the keynote announcement that Facebook open-sourced React Native on Android, Olivia provided more context on how React Native makes mobile app development easier using the popular React framework. Since its introduction in April on iOS, many developers have contributed to the React Native open source codebase, enabling several startups to quickly build and launch their apps.
Box: Building Mobile as a Platform-First Architecture
Mobile development at scale means shifting to reusable components. During this talk, attendees heard Martin discuss how Box transitioned its mobile strategy from building end user apps to focusing on mobile SDKs and mobile platform components.
Data Track
During the data track, engineers discussed challenges with scaling real-time stream and batch processing, data warehousing and analytics, monitoring, cache infrastructure, data serving, and storage systems.
Twitter Heron: Stream Processing at Scale
Heron has completely replaced Storm at Twitter. Karthik presented the challenges with Storm and remedies Heron introduced. Karthik focused on Heron’s design and how it compares with Storm on efficiency, latency, and throughput.
Pinterest: A Serving System for Batch-Generated Data Sets
During this talk, Pinterest announced that it is open-sourcing Terrapin. Terrapin attempts to localize data so that it can benefit from short-circuiting local reads. Before covering their chosen architecture, Varun discussed the trade-offs made: from Hadoop on HBase (problematic for large data sets) to bulk-loading HFiles (system ends up overloaded with compactions).
Microsoft: Experience/Learnings of Using Spark and Cassandra for Big Data Analytics
Arun Jayandra talked about how Microsoft built a solution for analytics using Spark and Cassandra, the challenges they faced, and key lessons learned.
Dev Tools Track
In the dev tools track, engineers focused on how to build efficient tools to increase production at scale. Key topics included bug analyzers, engine-query tools, and open source programs, as well as the results companies are seeing from deploying these systems.
How the Cloud Has Transformed Google Software Development
Google shared insights into how their team writes systems through continuous delivery, the role of DevOpps in efficient feature launches, and how Google’s engineering culture thrives. Google is making tools like the Cloud Debugger, Cloud Trace, Bazel, and ShipShape publicly available via the Google Cloud Platform.
Engineering Effectiveness at Twitter: Let 1,000 Flowers Bloom. Then Tear 999 of Them Out By the Roots.
Twitter has grown a lot — from a tiny company hosting its website on Ruby on Rails to a slightly larger company on the world’s largest Ruby on Rails app (the Monorail). The company now has over a thousand engineers and hosts a website and mobile apps built on hundreds of JVM-based services running in multiple data centers. Peter addressed the complex software development challenges and how the company got to where it is now.
The Motivation for a Monolithic Codebase: Why Google Stores Billions of Lines of Code in a Single Repository
Rachel shared the scale of Google’s codebase, including a description on Google’s custom-built monolithic source repository and the reasons this model of source control management works. She also shared background on Google’s systems and workflows that make managing and working productively with a large repository feasible.
Didn’t see what you were looking for? All tracks from @Scale 2015 can be seen here.
