As part of a team flushing out Facebook’s 2008 Election campaign designed to get more people to register to vote and then actually vote, I was responsible for the message on top of the News Feed. We had our design and messaging ready to go as the weekend before election day was approaching. The message was up and ready to be turned on. During the week, someone had suggested that we create an “I voted!” button that a user could click and tell his or her friends about the patriotic action of the day. I was excited; clearly this would encourage more people to go vote so they could tell their friends. So I buckled down, flushed out a News Feed and Minifeed story, and checked it in with plenty of time before Tuesday. On Sunday someone threw out an idea to have a counter that people could watch and see that their vote was indeed being counted on Facebook as well. The election team decided this would be awesome functionality during a quick chat on Monday morning. I had about 12 hours to design a counter that could handle millions of clicks and live update via ajax regularly enough to be exciting on millions of browsers at the same time. Here’s the last-second solution I came up with.
Every time a user clicked “I voted!”, I logged a message to our internal open-source logging system Scribe that included time, user id, and which U.S. State the user is in. I wrote a quick aggregation script that tailed the Scribe log. The script aggregated by state and globally. Then, it updated a memcached key that was accessible from our web tiers. The script ran throughout the day on my dev box, sending a heartbeat to our monitoring tools so it would email me if it died. Since the counter was live updating, millions of browsers would be asking for the same key over and over. So, I replicated the key 20 times across different memcached servers to not topple any single memcached box. With a couple designers, we made a cool fade animation that would update the counter every 10 seconds using our animation library to make it interesting to watch.
We pushed the counter Monday night with code to turn it on automatically and went to sleep. The next morning, I noticed the counter only updating every 30-40 seconds. While intriguing, most people wouldn’t see an update before they went to another page. It turns out that scribe was only flushing its logs to our mounts every 30 seconds. I made a quick change that forced the counter to update its number one-third of the difference between its shown value and the real value every seven seconds. Thus, it would show a number very close to the live value, but almost always have something to update. I committed, merged, and pushed by 10am.
The election page and counters came together quite quickly. It involved last-second commits, hotfixes, pushes, and scripts running on dev servers. We received over 5.4 million clicks and more than 3.6 billion counter update requests in under 24 hours. Thanks to Ben, Alex, Niket, and the rest of the election team for the great designs, ideas, and execution.
