RSA Hack Thoughts

I read the article “The Full Story of the Stunning RSA Hack Can Finally Be Told” by Andy Greenberg in Wired and several things stood out to me. So this is my commentary on the article and events that are covered.


“It opened my eyes to supply chain attacks.” says Mikko Hypponen, chief research officer at F-Secure, who worked with Hirvonen on the company’s analysis of the RSA breach.

While the RSA hack was certainly novel in one way, going after the cryptographic seeds to the two-factor authentication fobs, the bigger concept was certainly not new. Even in the late 80’s and early 90’s, the same concept played out many times. While vulnerabilities were prevalent and breaking into most machines was fairly easy, there were high-value targets that proved challenging. To achieve that, some hackers would target the vendors of the operating systems and break in there first. The primary targets were the bug databases where customers reported issues as well as the source code of the operating system. These two things could give them a huge advantage in compromising additional systems. Seeing this same strategy play out twenty years later should not have been that new to anyone that had been around during that timeframe.

“After 10 years of rampant state-sponsored hacking and supply chain hijacks, the RSA breach can now be seen as the harbinger of our current era of digital insecurity – and a lesson about how a determined adversary can undermine the things we trust most.”

After two decades of every type of organization in just about every country getting hacked, defaced, and data stolen, how exactly is this a ‘harbinger’? Governments, military, and security companies all popped for decades, but this one is a harbinger to what exactly? More organizations getting hacked?

A staffer in Australia had received an email with the subject line “2011 Recruitment plan” and an Excel spreadsheet attached to it. He’d opened it. Inside the file was a script that exploited a zero-day vulnerability – a secret, unpatched security flaw – in Adobe Flash, planting a common piece of malicious software called Poison Ivy on the victim’s machine.

This paragraph sums up the “Advanced Persistent Threat” (APT) that hacked RSA. Other than using a zero-day vulnerability in Flash, one of five that year, nothing really stood out about this from the attacker’s side.

A hacker wouldn’t have even been able to exploit the Flash vulnerability if the victim had been running a more recent version of Windows or Microsoft Office, or if he’d had limited access to install programs on his PC – as most security administrators for corporate and government networks recommend, Hirvonen says.

Correct. Microsoft acknowledged shortly after details were published that if RSA has been running a newer version of Microsoft Office, it would have come with protections that likely would have seriously limited the attack and/or added additional hurdles for them to then pivot into the network. At every point of this story it is important to remember that this is a security company. They received huge money to give advice like “patch often” and “upgrade to the latest version” while not doing it themselves.

In fact, several RSA executives came to believe that at least two groups of hackers were in their network simultaneously – one highly skilled group exploiting the other’s access, perhaps, with or without their knowledge.

I wonder if anyone suggested the piggyback could have been the TAO group at the NSA? We know that is a modus operandi of theirs while watching nation-state adversary hackers.

On that Australian employee’s PC, someone had used a tool that pulled credentials out of the machine’s memory and then reused those usernames and passwords to log into other machines on the network. They’d then scraped those computers’ memories for more usernames and passwords—finding some that belonged to more privileged administrators. The hackers eventually got to a server containing hundreds of users’ credentials. Today that credential-stealing hopscotching technique is common. But in 2011 the analysts were surprised to see how the hackers fanned out across the network.

Which analysts were surprised? This was standard operating procedure for hackers in the late 80’s and early 90’s. This is exactly what the hacking group I was in did. The only difference is back then you were likely to find trusted relationships and common passwords between vastly different networks (e.g. an .edu machine and a .gov or .mil).

RSA executives told me that the part of their network responsible for manufacturing the SecurID hardware tokens was protected by an “air gap”—a total disconnection of computers from any machine that touches the internet. But in fact, Leetham says, one server on RSA’s internet-connected network was linked, through a firewall that allowed no other connections, to the seed warehouse on the manufacturing side.

To be clear, RSA executives did not understand what “air-gapped” means, or were lying about it. I feel this is an important take-away.

Breaches as extensive as the one carried out against RSA are often discovered months after the fact, when the intruders are long gone or lying dormant. But Duane says that the 2011 incident was different: Within days, the investigators had essentially caught up to the intruders and were watching them in action.

“I basically shut off RSA’s business,” he says. “I crippled the company in order to stop any potential further release of data.”

One person in legal suggested they didn’t actually need to tell their customers, Sam Curry remembers.

The RSA staffers began putting in nearly 20-hour workdays, driven by the chilling knowledge that the breach they were tracking was still unfolding.

This seems like a lot of fluffing RSA over this hack, but ultimately this was the same company that didn’t patch their Windows boxes and didn’t air-gap the seeds like execs claimed. Among all of these positive mentions for $person doing the right thing, we always get “that one guy we will not name” for proposing absolutely shitty ideas or having a bad take. I understand they won’t throw anyone under the bus but this is an important dichotomy.

“Recently, our security systems identified an extremely sophisticated cyberattack in progress,” (RSA notification)

Which part was sophisticated? Exploiting unpatched machines, pivoting, or stealing data? When seemingly every attack is a “highly sophisticated cyber attack“, is it really highly sophisticated?

In fact, by the time Castignola had landed in Massachusetts, both the NSA and the FBI had been called to help the company’s investigation, as had defense contractor Northrop Grumman and incident response firm Mandiant. (By chance, employees of Mandiant had already been on-site prior to the breach, installing security sensor equipment on RSA’s network.)

How’d that Mandiant software work out for RSA I wonder? It doesn’t seemed to have helped prevent or detect the intrusion at any point based on the story told.

Multiple executives insisted that they did find hidden listening devices—though some were so old that their batteries were dead. It was never clear if those bugs had any relation to the breach.

Uh, this isn’t burying the lede, but it is burying a big one. I have so many questions and I don’t recall there being answers to this specific bit. How were there so many listening devices in RSA executive offices? Had they never done a single bug sweep? Were each of the devices found investigated? Did they do a sweep of all offices after finding them? If not, why not?

“Well it didn’t take long for whoever cracked RSA to find a lock to fit that key,” Cringely wrote. “What if every RSA token has been compromised, everywhere?” Two days later, Reuters revealed the name of the hacked military contractor: Lockheed Martin, a company that represented a cornucopia of ultra-secret plans for weapons and intelligence technologies. In the days that followed, defense contractors Northrop Grumman and L-3 were also named in news reports.

Today, with 10 years of hindsight, Coviello and other former RSA executives tell a story that starkly contradicts accounts from the time : Most of the former RSA staff who spoke to me claim that it was never proven that SecurID had any role in the Lockheed breach. Coviello, Curry, Castignola, and Duane all argued that it was never confirmed that the intruders inside RSA’s systems had successfully stolen the full list of seed values in an uncorrupted, unencrypted form, nor the customer list mapped to those seeds necessary to exploit them. “I don’t think that Lockheed’s attack was related to us at all,” Coviello states flatly.

vs A Lockheed source with knowledge of the company’s incident response reaffirmed to WIRED the company’s original claims. “We stand by our forensic investigation findings,” the source says. vs In a briefing to the Senate Armed Services Committee a year after the RSA breach, NSA’s director, General Keith Alexander, said that the RSA hack “led to at least one US defense contractor being victimized by actors wielding counterfeit credentials,” and that the Department of Defense had been forced to replace every RSA token it used.

Can we figure out who is trying to re-write history here? Sure seems like RSA is despite several other organizations saying otherwise. That would explain why so many were willing to go on the record for this article.

Adobe, Qualys, CVE, and Math

[This was originally published on the OSVDB blog.]

Elinor Mills wrote an article titled Firefox, Adobe top buggiest-software list. In it, she quotes Qualys as providing vulnerability statistics for Mozilla, Adobe and others. Qualys states:

The number of vulnerabilities in Adobe programs rose from 14 last year to 45 this year, while those in Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.

This caught my attention immediately, as I know I have mangled more than 45 Adobe entries this year.

First, the “number of vulnerabilities” game will always have wiggle room, which has been discussed before. A big factor for statistic discrepancy when using public databases is the level of abstraction. CVE tends to bunch up vulnerabilities in a single CVE, where OSVDB tends to break them out. Over the past year, X-Force and BID have started abstracting more and more as well.

Either way, Qualys cited their source, NVD, which is entirely based on CVE. How they got 45 vulns in “Adobe programs” baffles me. My count says 97 Adobe vulns, 95 of them have CVEs assigned to them (covered by a total of 93 CVEs). OSVDB abstracted the entries like CVE did for the most part, but split out CVE-2009-1872 as distinct XSS vulnerabilities. OSVDB also has two entries that do not have CVE, 55820 and 56281.

Where did Qualys get 45 if they are using the same CVE data set OSVDB does? This discrepancy has nothing to do with abstraction, so something else appears to be going on. Doing a few more searches, I believe I figured it out. Searching OSVDB for “Adobe Reader” in 2009 yields 44 entries, one off from their cited 45. That could be easily explained as OSVDB also has 9 “Adobe Multiple Products” entries that could cover Reader as well. This may in turn be a breakdown where Qualys or Mills did not specify “Adobe Software” (cumulative, all software they release) versus “Adobe Reader” or some other specific software they release.

Qualys tallied 102 vulnerabilities that were found in Firefox this year, up from 90 last year.

What is certainly a discrepancy due to abstraction, OSVDB has 74 vulnerabilities specific to Mozilla Firefox (two without CVE), 11 for “Mozilla Multiple Browsers” (Firefox, Seamonkey, etc) and 81 for “Mozilla Multiple Products” (Firefox, Thunderbird, etc). While my numbers are somewhat anecdotal, because I cannot remember every single entry, I can say that most of the ‘multiple’ vulnerabilities include Firefox. That means OSVDB tracked as many as, but possibly less than, 166 vulnerabilities in Firefox.

Microsoft software dropped from 44 to 41, according to Qualys. Internet Explorer, Windows Media Player and Microsoft Office together had 30 vulnerabilities.

According to my searches on OSVDB, we get the following numbers:

  • 234 vulnerabilities in Microsoft, only 4 without CVE
  • 50 vulnerabilities in MSIE, all with CVE
  • 4 vulnerabilities in Windows Media Player, 1 without CVE
  • 52 vulnerabilities in Office, all with CVE. (based on “Office” being Excel, Powerpoint, Word and Outlook.
  • 92 vulnerabilities in Windows, only 2 without CVE

When dealing with vulnerability numbers and statistics, like anything else, it’s all about qualifying your numbers. Saying “Adobe Software” is different than “Adobe Acrobat” or “Adobe Reader” as the software installation base is drastically different. Given the different levels of abstraction in VDBs, it is also equally important to qualify what “a vulnerability” (singular) is. Where CVE/NVD will group several vulnerabilities in one identifier, other databases may abstract and assign unique identifiers to each distinct vulnerability.

Qualys, since you provided the stats to CNet, could you clarify?

Who’s to blame? The hazard of “0-day”.

[This was originally published on the OSVDB blog.]

This blog entry is probably worth many pages of ranting, examining and dissecting the anatomy of a 0-day panic and the resulting fallout. Since this tends to happen more often than some of us care to stomach, I’ll touch on the major points and be liberal in pointing fingers. If you receive the “wag of my finger“, stop being part of the problem and wise up.

I blinked and missed someone disclosing that there was a dreaded 0-day vulnerability in Adobe Flash Player and that it was a big threat. Apparently Symantec noticed that evil Chinese sites were exploiting Flash and the current 9.0.124.0 could be successfully exploited. When pressed for details, Symantec backtracked and said that they were wrong and it appeared to be the same exploit as previously disclosed by Mark Dowd (CVE-2007-0071). Bad Symantec, poor research.

To make matters worse, Symantec then further claimed that even though it was an old issue, the “in-the-wild exploit was effective against stand-alone versions of Flash Player 9.0.124.0” and that not all versions had been patched correctly. Way to save face Ben Greenbaum of Symantec!! Oh wait, today he changed his mind and said that Symantec’s claims were based on erroneous conclusions and that the behavior of Flash on Linux they were observing was indeed intended by Adobe and not proof it was vulnerable. To make matters worse, Symantec researchers downloaded the “latest” Flash and found it “vulnerable”, which lead to their sky-is-falling panic. Shortly after, they realized that they didn’t download all of the security patches and had been exploiting a known vulnerable version of Flash. Oops?

Two rounds of hype-driven 0-day threat warnings, and no real new threat. Whew, hopefully Symantec raised their THREATCON to blood red or whatever is appropriate for such 0-day threats. You do monitor that don’t you?

This fiasco lead many news outlets and vendors to issue warnings about the new 0-day threat. Secunia, SecurityFocus/BID, SecurityTracker, CERT, and FrSIRT all released new warnings and created entries in their respective databases as a result. In the VDB world, this is a royal pain-in-the-ass to deal with. Secunia ‘revoked’ their entry, BID ‘retired’ their entry, SecurityTracker flaged theirs ‘duplicate entry’, FrSIRT ‘revoked’ their entry and CERT still has it listed.

Fortunately for OSVDB, we were a few hours behind the rest and noticed the discrepancies and waited for more information. Unfortunately, the rest of the world, including ALL of the VDBs and news outlets listed above (and others) failed miserably in using common sense and a government funded resource to better prevent this kind of problem. As of this posting, Secunia, BID, SecurityTracker, FrSIRT, CERT, Dancho, ComputerWorld and eWeek still don’t link to the CVE ID for the vulnerability. Only Adobe’s updated blog entry actually references CVE-2007-0071 (but doesn’t link to it). Secunia links to a previous ID that has seven CVEs associated with it. The original CVE was assigned 2007-01-04 and published around 2008-04-08, a month and a half prior to this mess.

VDBs, shame on you for adding to the confusion. Symantec, shame on you for crying 0-day when your own engineers screwed up badly. Shame on everyone for not clearing it up fully by linking to the correct CVE entry or their own previous entries.

Before any of you receiving a “wave of the finger” bitch, consider the real world impact of your actions. In this case, only 12 MILLION people ended up seeing a vague warning when they loaded their favorite game. Blizzard included the correct fix information which was the same as a month or more before, but the sudden ‘security alert’ (that is extremely rare) only prompted their customers to wonder, possibly panic and definitely kill some demons as a result.