Buffer’s April Engineering Report: New Happiness Dashboard, New iOS App and 20+ Vulnerabilities Patched

Coming off of the momentum we had in March, April was another solid month for the engineering team at Buffer.  We really stepped up our game on the security front.  We’ve also made strides to better scale Buffer, overhauled the ios7 app, and continued making progress with hiring.

Here’s a tl;dr of how April looked

• Over 20 security vulnerabilities were patched and we released some awesome security features
• No systemwide downtime (win!) but still had a few hiccups
• 1 new offer made, and 6 awesome engineers were interviewed
• Made significant steps to sharding our database
• Now sending 2.7 million updates a week
• Started a Happiness Dashboard written in NodeJS and Coffeescript!
• Struggled with blogging and open source
• Deployed our new iOS7 version of Buffer and released 2 Android updates

Retreat projects: All hands on deck

The beginning of April was incredible. We had the entire team working out of the same room for an entire week in Cape Town, South Africa. This was the first time we had more than 3 engineers together! It was incredible what we accomplished in 5 days.

• Colin worked on the beginning of the Happiness Dashboard. The Happiness dashboard is a tool that we’re using to gauge how quickly we respond to our awesome customers on Olark, Twitspark, and Helpscout. The plan is for this to be public so the whole world can see!
• Andy focused on the final touches to our iOS7 app. We released this in early April and it’s had such an amazing response!
• Steven and Joel worked together to overhaul our growth dashboard. Steven and I also re-architected the dashboard so that we can display past data and a way to generate A/B test results in real-time.
• Dan focused on a much better way to set up experiments and handle the grandfathering of user features and built out our API proxy for the web so that web access tokens are better protected.
• Niel built our new feeds Buffer feature with Zach building the backend.

Security: More than 20 vulnerabilities patched

A big change we made in April was to separate out the security tickets of our bounty program to a separate inbox that’s managed by the engineers. This was such a good move since now we get Hipchat notifications anytime a new inquiry gets in. We’ve moved much faster tackling vulnerability reports.

In April we fixed up over 20 various vulnerabilities! Creating a bounty program was really one of the best decisions we’ve made. It’s amazing to see how quickly some of the vulnerabilities that get introduced by developers are reported.

Two-step sign-in for Android and iOS

We were pretty excited to finally release two-step sign-in on Android and iOS. This was a huge hole for us, and we’re so glad to finally close that one up.

Session controls within the app

I’m also super excited that we’ve built out session controls within the app. Now all Buffer users have a way to display the past logged-in sessions and can revoke any current sessions. Just head over to the My Account > Access and Password section to check this out.

Heartbleed

The beginning of April was a bit rough. Like many others, we were vulnerable to the Heartbleed bug in OpenSSL. This happened during the retreat. We made sure to jump on this quickly and on the same day we noticed this, we worked with AWS to patch our servers, revoked our SSL certificate and re-issued a new one. Here’s the blog post that we wrote to keep everyone in the loop during that time!

Reliability and scaling

I’m so happy to report that we had a solid month without any trouble system wide. There were no issues in posting status updates or adding them to the Buffer :). With that said, we still weren’t without issues.

We had a major issue early in April in which our buff.ly domain was suspended by our registrar when they identified the domain as creating spam. After some investigation we noticed there has been a malicious Buffer user who would shorten links with buff.ly and send email phishing emails. After working with our registrar we got buff.ly back online. This wasn’t at all ideal and it took us two days to fully bring buff.ly online and to resolve to the right IP. We took this opportunity to switch nameservers to AWS Route53 so that we have more control over this. We’re also actively tracing down malicious users taking advantage of our shortening.

In April we continued expanding our monitoring tools. We added New Relic server monitoring and created more alerts to have more details about the health of our platform.

Hiring

Early on in April, hiring took a backseat for us as we focused on the product and security while on the retreat. After coming back from the retreat and getting into the regular swing, we caught up.  Six engineers interviewed in April, and one offer was made.

Blogging and open source

April was not a great month for us in progressing with our goals of blogging and open source. Unfortunately we haven’t blogged much on the engineering side in the past few weeks, nor did we make much headway with open source. It’s possible the traveling and focus with the retreat may have had an impact on getting into a better groove, however that’s not an excuse. I’m brainstorming ways we can make this a higher priority in May. I’m looking forward to open sourcing our Android app, hopefully we can do this in May!