Buffer security breach has been resolved – here is what you need to know

Update: This article was originally titled “Buffer has been hacked – here is what’s going on”. The hacking incident happened yesterday (Saturday) and below is a recap of everything that happened. Please ask us any questions you have in the comments below.

If you’re reading this, the most important section for you is Update 7.

We’ve discovered the source of the breach and closed the vulnerability. Keep reading for the full story.

Update 9: We’ve discovered the exact details of how the Twitter and Facebook access tokens were obtained to send spam posts.

Update 10: We’ve uncovered this weekend that the hackers also gained access to our code.

I wanted to post a quick update and apologize for the awful experience we’ve caused many of you on your weekend. Buffer was hacked around 2 hours ago, and many of you may have experienced spam posts sent from you via Buffer. I can only understand how angry and disappointed you must be right now.

Not everyone who has signed up for Buffer has been affected, but you may want to check on your accounts. We’re working hard to fix this problem right now and we’re expecting to have everything back to normal shortly.

We’re posting continual updates on the Buffer Facebook page and the Buffer Twitter page to keep you in the loop on everything.

The best steps for you to take right now and important information for you:

• Remove any postings from your Facebook page or Twitter page that look like spam
• Keep an eye on Buffer’s Twitter page and Facebook page
• Your Buffer passwords are not affected
• No billing or payment information was affected or exposed
• Update: All FB posts are being posted normally again and no more spam postings will occur.

I am incredibly sorry this has happened and affected you and your company. We’re working around the clock right now to get this resolved and we’ll continue to post updates on Facebook and Twitter.

If you have any questions at all, please ask in the comments below or email us hello@bufferapp.com. Understandably, a lot of people have emailed us, so we might take a short while to get back to everyone, but we will respond to every single email.

We’re going to update this article as we’ve got more news to share for you!

Update: 1pm PST

All hidden Facebook posts are now shown again. There is a chance that some spam postings are now live again, please check on your Facebook accounts and delete them.

No more spam updates should occur at this point, as all posting has been disabled. All hidden posts however should show again.

We’re keeping you posted on everything!

Update 2: 3pm PST

We’ve increased security for how store Twitter tokens and deployed a fix.

You can login with Twitter again. You will have to reconnect all your Twitter accounts in Buffer. Here is how to reconnect them.

You can now send Tweets via Buffer again.

Update 3: 5:30pm PST

We’re currently working directly with Facebook and AWS to get this all sorted out. It looks like we are making our way closer to a full recovery. Twitter (see Update 2 above) should be working again 100%.

About your Facebook posts: Currently it’s  not possible to connect or post to FB with Buffer. We hope to have this working again real soon for you and I greatly apologize for the hassles this might have created.

Update 4: 8pm PST – All posting is working again!

We’ve greatly increased the security of how we handle all social messages being posted and everything is back to normal. Please try signing into your Buffer account from http://bufferapp.com instead of the mobile apps for now.

For your Facebook account:

If you had Facebook posts via Buffer scheduled during the outage, they will likely appear as “failed” in your Buffer queue. You can just hit “retry in Buffer” and they should then be scheduled normally and go out as expected again.

For your Twitter account: You will have to reconnect all your Twitter accounts in order to start posting again. Here is how you can reconnect your Twitter account.

We’re also going to publish an in-depth post about what the spammers got access to and what we did to fix it. In short, we encrypted all access tokens for Twitter and Facebook and also added other security measurements to make everything much more bullet proof. More on this in a coming post!

Update 5: 9:00am PST Sunday, 27th of October

We have monitored all behaviour overnight and everything has remained normal. All posts to Facebook and Twitter via Buffer should be going out normally. For Twitter you will have to reconnect your accounts from the web dashboard.

We have greatly increased security of how we are posting to Twitter and Facebook and have confidence to cover the security holes the hackers have used to break into our system.

What’s next: We’re working with several security experts on tracking down exactly how it was possible for the spammers to get into our system. We’re making good progress on this, this morning. What will follow is that we’re going to publish an in-depth update on the impact of the hack and everything we know about how it happened.

Update 6: 3:00pm PST Sunday, 27th of October – Extent of the impact

We’re now able to recover further insights as to what has happened exactly:

As soon as we noticed the issue, we disabled all postings to prevent more spam from going out.

In terms of exact numbers, Facebook confirmed with us that 30,000 Buffer users who had a Facebook page connected (out of  476,343 total connected pages to Buffer) were affected and had spam posted on their behalf. This means that 6.3% of Buffer users on Facebook were impacted by this.

Since then we’ve taken key security measures: we have added encryption of OAuth access tokens and we have changed all API calls to use an added security parameter.

Service has resumed with increased security since the incidents. You can now head into the Buffer dashboard and use Buffer again as normal.

Update 7: 8:00pm PST Sunday, 27th of October – All Twitter accounts need to be reconnected

We’ve taken further security measures and as a result all Twitter accounts will have to be reconnected. Even if you’ve already done so, you will have to reconnect your account one last time.

I greatly apologize for having you do this again, but we want to make sure that we are on the safe side with this.

Go to the web dashboard now to reconnect all your Twitter accounts.

Update 8: 3:00pm PST Monday, 28th of October – The origin of the breach has been identified

Here is an important note from our CTO Sunil, who is leading the technical investigation on this issue:

We’ve learned how the hackers breached our system on Saturday. We’ve worked with our partners to trace back their steps and we’ve closed the vulnerability. This is a big relief, as understanding how the hack occurred was the biggest worry in our eyes. Here’s what we know.:

• The hackers were able to steal some of our Facebook and Twitter access tokens from our users. We have confirmed that the hackers were not able to get access to any passwords, billing information or other user information other than specifically the Twitter and Facebook access tokens.
• We have since invalidated all Twitter access tokens. We’ve added encryption for all Twitter access tokens.
• For Facebook API calls we are now using an extra security parameter to make all tokens more secure.
• With these improvements your Twitter and Facebook accounts are not at risk anymore. Attackers will not be able to use this method to send spam anymore.
• The method which left our data vulnerable is now locked and secure.

My apologies for the disruptions this has caused you and your company. Please feel free to ask me any questions about this below. We are going to add more detailed updates about this as we uncover more. Expect a detailed report!

Update 9: 2:00pm PST Tuesday, 29th of October – How were Twitter and Facebook access tokens obtained to spam?

As of today, we’ve learnt some important, new information about how the hackers were able to get access to the Buffer database and steal the API tokens for Twitter and Facebook that were used to post spam on our users behalf.

The backdoor that was created through one of our partners, MongoHQ who are managing our database. MongoHQ, who have been incredibly responsible and responsive regarding this also just released an update about the security breach on their blog.

In short, the MongoHQ password of one of MongoHQ’s employees was stolen. That way the hackers logged into the main admin dashboard of MongoHQ and were able to use the “impersonate” feature to see all of Buffer’s database information. Through that, they wrote a script to steal our social access tokens and post spam messages on behalf of our users.

From their blog post:

“On October 28, our operations team detected unauthorized access to an internal, employee-facing support application.

We immediately responded to this event, by shutting down our employee support applications and beginning an investigation which quickly isolated the improperly secured account. We have determined that the unauthorized access was enabled by a credential that had been shared with a compromised personal account.

No internal application was made available to our team before a team-wide credential reset and audit.

Users of our support application have access to account information, including lists of databases, email addresses, and bcrypt-hashed user credentials.

Our support tool includes an “impersonate” feature that enables MongoHQ employees to access our primary web UI as if they were a logged in customer, for use in troubleshooting customer problems. This feature was used with a small number of customer web UI accounts. Our primary web UI allows customers to browse data and manage their databases. We are contacting affected customers directly.”

We have full trust in MongoHQ that they have closed the security hole and are also very grateful about their fast update and the company helping us clear up all confusions in connection to the breach.

I want to be clear that this is still our fault. If access tokens were encrypted (which they are now) then this would have been avoided. In addition, MongoHQ have provided great insights and have much more logging in place than we have ourselves. We’re also increasing logging significantly as a result.

Update 10: 1:00pm PST Monday, 4th of November – Hackers also gained access to Buffer’s code via GitHub

We’ve recently (more precisely on Friday night) learned some important new information that I want to share with you.

We’ve been continuing investigation since the initial spam attack. We’ve learned that in addition to extracting access tokens through our database provider MongoHQ, the hackers were also able to get unauthorized access to our Buffer GitHub account. We discovered this by viewing the Security History of one of our team members whose account was used to access the code. The logs showed logins and current sessions, which we revoked immediately. With access to the code they extracted our consumer key and secret to post spam to Twitter.

We suspect that the way this happened could have been through the Adobe leaked passwords or some other potential leak where a large number of passwords were stolen. We have no way of proving this.

After learning about this, we secured the codebase and then reset our Twitter consumer key and secret once more. It seems clear by now that Buffer was specifically targeted for this spam attack.

The whole Buffer team has changed their passwords and enabled 2-step login for as many services as allow it. (Google, AWS, Twitter, Facebook, Github).

We are going to publish more info on this as we learn more.

Please ask us any questions in the comments below or if you have any questions just email us hello@bufferapp.com.

We’re keeping you posted about all new updates from here!