The Buffer February Engineering Report

Sunil Sadasivan -

February was an exiting time for the engineering team at Buffer. Here are some of the numbers at a high level.

  • 2 new engineers joined the team (1 Growth Engineer, 1 Front-end)
  • 10,243,834 updates were posted to social networks in February
  • 0 mins of system wide outage (woohoo!)
  • 10 minutes of web dashboard outage
  • 405 ms: Our average api response time

SSL and other security improvements

We continued with our momentum with our security improvements in February. The Hall of Fame program has really pushed us to make several improvements throughout our app. In February we made the switch so that the web dashboard is fully served by SSL only. In doing this, we’ve also ensured that our session cookie securely transported.

We fixed several vulnerabilities reported by awesome whitehat hackers. When we opened our security Hall of Fame in January, it brought in more than 55 emails in February and we moved fast to fix up many of these.

Downtime and reliability

Februrary was a solid month in terms of platform reliability. We didn’t have any systemwide outages, which is always good. This means that updates scheduled were posted on time and there was no downtime in using our API.

On Wednesday, Feb. 5, we introduced the switch to force SSL for all of our pages. When we did this, we experienced 10 minutes of web down time. This was due to an oversight in making the switch. Our load balancer health check started failing because it was checking the application health on port 80. Since it now receives a 301 redirect, it indicated this as a failure and this brought down all of our servers as unhealthy. After quickly realizing this is what happened, we reverted the change and made appropriate adjustments. We had a solid postmortem, and overall this was a great learning experince.

Introducing Bufferbot

In February, one of the biggest changes we made to our development flow was introducing our Jenkins CI server. For a while, our unit test suite would run on our local machines before deploying a change. This added delays to deploying and required fast internet which ended up being a huge bottleneck. In February we set up ‘Bufferbot,’ our Jenkins continuous integration server which triggers off the running of our test suites every time a repository push is made to a remote branch. We integrated this with our chat tool, Hipchat and now our tests run asynchronous to our deploys. We’ve also closely integrated Bufferbot with our Github repositories so that if a build fails, this is indicated on a pull request so that we don’t accidentally merge a broken branch.

Analytics, weekly digests and scaling challenges

We accomplished quite a lot on the engineering side in terms of features. Zach built out our real-time analytics for business customers using Twitter Sitestreams. Now analytics for Twitter profiles for everyone on business plans are updated in real-time. This was definitely a fun and challenging task to ensure this scales well.

In February we also launched weekly digests. This was another interesting challenge as we now send out over 600,000 emails of weekly statistics in the span of 3 hours on Monday mornings. Colin was the master of this task and in charge of building out and load testing to ensure this doesn’t impact our other services. Implementing this task was certainly not trivial.

Niel was focused on profile grouping and extension/composer re-architecting and we’re now testing this internally :).

Andy‘s been hard at work finishing up changes for IOS7. In February, he released the Buffer for iPhone v3.0 beta to a select number of Buffer users for testing and feedback. We’ll be ready for submission in the next few weeks.

I’m writing this post two weeks into March, and so I’m already excited to get started on March’s update. Here’s a hint of some of the things coming in our next report: More engineering blog posts, developing a process for postmortems with the “5 whys,” a new focus on testing, and several core platform improvements and performance improvements.

Come join us

Every day is an adventure working on the engineering team at Buffer. There’s a lot of excitement when you’re trusted with scheduling more than 10 million social media posts a month. Interested in helping us out? Come join our team!