Twitter | Search | |
Colm MacCárthaigh
I think right around this minute is just about exactly 5 years since the Heartbleed vulnerability in OpenSSL became public. I remember the day vividly, and if you're interested, allow me to tell you about how the day, and the subsequent months, and years unfolded ...
Reply Retweet Like More
Colm MacCárthaigh Apr 7
Replying to @AWS
Five years ago I was the Principal Engineer for Elastic Load Balancer. I was about a year into that, having moved over to build some cool tech that would later become AWS HyperPlane. Previously I'd worked on CloudFront and Route 53 and DDOS stuff.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
CloudFront and ELB are easily two of the biggest TLS/SSL things at Amazon, and I'd previously worked on OpenSSL things, like Apache's mod_ssl, so then the issue went public ... I was one of the first people paged. I was on the 14th floor of our Blackfoot building.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
It was very very quickly evident that Heartbleed wasn't like other vulnerabilities. Normally there's a window between going public and exploits being crafted, but heartbleed was so easy to exploit that it took just minutes of poking around.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
Heartbleed was a memory disclosure vulnerability, which in theory is supposed to be less significant than a remote execution vulnerability, but this was scarier than any bug I'd ever seen. XKCD has an explainer ...
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
The TLS protocol had been extended to include a "Heartbeat" extension. It was intended for keep-alives and MTU discovery for DTLS, which uses UDP, but OpenSSL had included it in regular TLS too (which uses TCP).
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
And at bottom, the bug was simple, you send a small amount of data, and ask the server to send you back up to 16k of data, and it would send back 16K of decrypted plaintext from memory. URLs, passwords, cookies, credit cards, just about anything could be in there. Ouch ouch.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
OpenSSL was and is very very widely used, just about everyone was impacted in some way. AWS services, our competitors services, basically all of our customers in their own stacks. It felt like the internet was on fire.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
At Amazon we use conference calls for high severity events, usually operational, this was declared a security sev-1 (I've never seen another like this). Call leader that day was Kevin Miller. He just happened to be at all, but it worked out well because he had crypto experience.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
We quickly figured that we'd be patching everything that day, so an emergency was declared and all AWS software deployments were paused. This is incredibly disruptive, but the call leader has the authority to do this on their own. Our CEO and SVP agreed with the call.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
Within Amazon, we have our own package system called Brazil. At the time a part of (retail) owned our internal OpenSSL package, but over on ELB we took it over that day and came up with a minimal 2-line hot-patch. Didn't want new risks.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
Within about an hour, deployments with the hot patch were in progress, and it went out quicker than I've seen anything. Within a matter of hours, AWS was 100% patched. Even 5 years ago, this was millions of deployments. Amazingly, there were no reports of customer impact either.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
In parallel to that were discussions about customer messaging and notification. We were asked to analyze if we thought private keys could have been disclosed. This wasn't an easy call. It looked like keys weren't leaking, but intermediate data used as part of key operations was.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
My best guess on the day was that enough material was in there that keys could be at risk. I recommended thatl customers rotate and revoke keys if they can, and our CISO and CEO took that as good enough and began that painful process.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
About a week later, that hunch was proved right, we know for sure because CloudFlare ran a contest to see if folks could re-assemble keys and they could. Impressive stuff!
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
To backtrack a little: once the HeartBleed website went live (which incidentally was hosted on AWS S3! and there was never event a hint of taking it down) we started getting a *lot* of customer contacts.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
HeartBleed was really well marketed, which is a good thing! Months later in a presentation I showed that it made more headlines and news articles in one day than any war had since Vietnam. Good because people patched. 98% of customers patched within a week.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
I know that because on the night of Heartbleed we did something we never did before: we started vulnerability scanning every EC2 IP address and sending customers notifications. We thought it was a big enough deal that the emails would be worth it.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @pzb
The day after Heartbleed, our core cryptography people met, I remember was there, and we did a few more things with the OpenSSL package. Amazon's OpenSSL has always been a bit different than the public one, but that day we created a new "hardened" branch.
Reply Retweet Like
Colm MacCárthaigh Apr 7
Replying to @colmmacc
I won't go into what we did with it here, but quite a bit at the time, Emilia Kasper included some of the changes into base OpenSSL later I think. Our customers mostly upgraded to the latest public version from OpenSSL, which we had in Amazon Linux too.
Reply Retweet Like