Buying Into the Bias: Why Vulnerability Statistics Suck

Steve Christey, the CVE Editor from MITRE, and I gave a presentation at Black Hat Briefings 2013 on the problems we have witnessed over the years with poor vulnerability statistics. Rather than just debunk a handful, which we did, we also went into extensive detail on the different types of bias that ultimately lead to these bad stats. In addition to showing concrete examples of how the bias plays out, and how a single researcher can significantly impact stats, we also point out examples of ‘good-ish’ stats, since we haven’t seen truly good ones yet. Why? The data sources are so primitive, but they are all we have right now. In addition to the the slides presented, we left in over 40 additional working slides that didn’t make the cut. As always, there is additional commentary, references, and notes in the PPT that weren’t seen in the presentation.

Seriously RIM? Call it the HackBerry from now on…

[This was originally posted on the OSVDB blog.]

Our sponsor Risk Based Security (RBS) posted an interesting blog this morning about Research In Motion (RIM), creator of the BlackBerry device. The behavior outlined in the blog, and from the original blog by Frank Rieger is shocking to say the least. In addition to the vulnerability outlined, potentially sending credentials in cleartext, this begs the question of legality. Quickly skimming the BlackBerry enterprise end-user license agreement (EULA), there doesn’t appear to be any warning that the credentials are transmitted back to RIM, or that they will authenticate to your mail server.

If the EULA does not contain explicit wording that outlines this behavior, it begs the question of the legality of RIM’s actions. Regardless of their intention, wether trying to claim that it is covered in the EULA or making it easier to use their device, this activity is inexcusable. Without permission, unauthorized possession of authentication credentials is a violation of Title 18 USC § 1030 law, section (a)(2)(C) and potentially others depending on the purpose of the computer. Since the server doing this resides in Canada, RIM may be subject to Canadian law and their activity appears to violate Section 342.1 (d). Given the U.S. government’s adoption of BlackBerry devices, if RIM is authenticating to U.S. government servers during this process, this could get really messy.

Any time a user performs an action that would result in sharing that type of information, with any third party, the device or application should give explicit warning and require the user to not only opt-in, but confirm their choice. No exceptions.

Stalking me in Las Vegas…

dc-21-logo-sm

I fly out to Las Vegas tomorrow for the trifecta of summer security conventions held in oppressing heat. BlackHat Briefings, BSides Las Vegas, and DEF CON 21. If you want to catch up to talk about attrition.org, OSVDB, or anything vulnerability related, look for the disgruntled person likely wearing a squirrel-themed shirt. If you would like to stalk me down to catch up, chat about anything, or shank me, this friendly guide will assist you:

Wednesday

I will be at BSides in the morning to catch a few talks, mingle, and generally harass the BSides staff. Because they didn’t have enough going on putting together the entire convention. In the early afternoon I will make my way to BlackHat to register, visit a few vendor booths, and then give a presentation at 3:30 with Steve Christey in Palace 1 room. The talk is called “Buying Into the Bias: Why Vulnerability Statistics Suck”. Hopefully we will demonstrate how vulnerability statistics have sucked throughout the years, ways to improve them, and more. After the talk and any Q&A, I hope to stick around for the Pwnie awards and BarCon, before heading to either the Adobe or Tenable party.

Thursday

I will be at BSides almost all day and hope to catch a variety of talks that sound interesting. Later in the evening I will be in various places for a dinner meeting, and then may swing by the Microsoft party to harass their security people before ending up back at BSides for the after party.

Friday

I will be at Defcon and Skytalks, likely lingering around the Skytalks room if nothing else going on. At 8P, the Defcon Documentary is showing, as well as Hacker Pyramid / Hacker Jeopardy.

Saturday

I will be at Defcon and Skytalks, until 3P where I will present the Defcon Recognize Awards with Russ in Track 3. Come see who the charlatan of the year is, among other categories! That evening at 8P is the screening of the new movie “Reality Hackers” in Track 2. After that, probably doing vile things at the 303 party.

Sunday

If you haven’t found me by this time, you failed.

Cybercrime Stats: From Bad to Bad

[This was originally published on the OSVDB blog.]

Since vulnerabilities are a cornerstone of computer crime, stats on it are of interest to us. Statistics on cybercrime have always been dodgy; more so than real-world crime statistics. When your car is broken into or stolen, you know it. More often than not, you will report it to the police. In the computer world, an un-measurable number of intrusions happen every day. The rate of malware infection, DoS attacks, and other virtual crimes are not only difficult impossible to measure, they potentially go unreported more often than not.

Classically, the only number thrown around regarding damages from cybercrime has been this mythical one trillion dollars. Yes, with a ‘T’, not a ‘B’. That number has been challenged by many in the past, but no one has offered a better number with anything remotely factual. On July 22 the Center for Strategic and International Studies released a new study commissioned by McAfee (who previously quoted the trillion dollar figure) saying that damages are much less. From a Los Angeles Times article on the release:

Cyberattacks may be draining as much as $140 billion and half a million jobs from the U.S. economy each year, according to a new study that splashes water on a previous estimate of $1 trillion in annual losses.

“That’s our best guess,” said James Andrew Lewis, the director of the technology and public policy program at the Center for Strategic and International Studies.

James Andrew Lewis’ comment calling it a “best guess” is not assuring. The one trillion dollar figure cited for all those years was no better than a guess, as the surveys it relied on were far from a solid methodology. Regardless, the figure of $140 billion seems more rationale on the surface. Contrasting that is the claim that half a million jobs are “drained” from the U.S. economy each year. How can cybercrime conceivably lead to that? Reading on in the article:

Lewis and co-author Stewart Baker, a distinguished visiting fellow at CSIS, said that they were still working to determine cybercrime’s impact on innovation. They suggested a follow-up report might come out with a bigger number.

But preliminarily, they found U.S. losses to be somewhere between $20 billion to $140 billion, or about 1% of the nation’s GDP. They pegged job losses at 508,000.

“The effect of the net loss of jobs could be small, but if a good portion of these jobs were high-end manufacturing jobs that moved overseas because of intellectual property losses, the effect could be wide ranging,” Lewis said.

Right after the hint of a more rational number, CSIS immediately makes it a worthless number when they say it is really somewhere between $20 billion and $140 billion. In the world of sanity and statistics, that range is unreasonable. Further, Lewis goes on to say that some of the 508,000 jobs lost are due to “high-end manufacturing jobs moved overseas because of intellectual property losses”. Huh? High-end manufacturing jobs are moving overseas because of corporate budgets more than cybercrime. Such a claim should be backed up by a long list of examples showing companies losing intellectual property, and then reporting it to law enforcement or their shareholders, as well as SEC filings.

We moved from the fictional trillion number, to an overly wide range in the tens or hundreds of billions, and got an odd claim of half a million jobs lost due to cybercrime. This new study did little to clear things up.

Stilgherrian makes another great point in his ZDNet piece, that everyone should take to heart when reading cybercrime statistics:

If we’re killing one cybercrime myth, let’s kill another — one which coincidentally emerged from McAfee — namely that the wealth transfer due to hacking represents some historically-unprecedented economic disaster.

Ultimately, we also have to remember that any cybercrime statistics coming from a company like McAfee are suspect, as they directly benefit them while they sell computer security solutions.

Android versus iOS Security – Not Again…

[This was originally published on the OSVDB blog.]

About two weeks ago, another round of vulnerability stats got passed around. Like others before, it claims to use CVE to compare Apple iOS versus Android in an attempt to establish which is more secure based on “vulnerability counts”. The statistics put forth are basically meaningless, because like most people using a VDB to generate stats, they don’t fully understand their data source. This is one type of bias that enters the picture when generating statistics, and one of many points Steve Christey (MITRE/CVE) and I will be making next week at BlackHat (Wednesday afternoon).

As with other vulnerability statistics, I will debunk the latest by showing why the conclusions are not based on a solid understanding of vulnerabilities, or vulnerability data sources. The post is published on The Verge, written by ‘Mechanicix’. The results match last year’s Symantec Internet Security Threat Report (as mentioned in the comments), as well as the results published this year by Sourcefire in their paper titled “25 Years of Security Vulns“. In all three cases, they use the same data set (CVE), and do the same rudimentary counting to reach their results.

The gist of the finding is that Apple iOS is considerably less secure than Android, as iOS had 238 reported vulnerabilities versus the 27 reported in Android, based on CVE and illustrated through CVEdetails.com.

Total iOS Vulnerabilities 2007-2013: 238
Total Android Vulnerabilities 2009-2013: 27

Keeping in mind those numbers, if you look at the CVE entries that are included, a number of problems are obvious:

  1. We see that the comparison timeframes differ by two years. There are at least 3 vulnerabilities in Android SDK reported before 2009, two of which have CVEs (CVE-2008-0985 and CVE-2008-0986).
  2. These totals are based on CVE identifiers, which does not necessarily reflect a 1-to-1 vulnerability mapping, as they document. You absolutely cannot count CVE as a substitute for vulnerabilities, they are not the same.
  3. The vulnerability totals are incorrect due to using CVE, a data source that has serious gaps in coverage. For example, OSVDB has 71 documented vulnerabilities for Android, and we do not make any claims that our coverage is complete.
  4. The iOS results include vulnerabilities in WebKit, the framework iOS Safari uses. This is problematic for several reasons.
    1. First, that means Mechanicix is now comparing the Android OS to the iOS operating system and applications.
    2. Second, WebKit vulnerabilities account for 109 of the CVE results, almost half of the total reported.
    3. Third, if they did count WebKit intentionally then the numbers are way off as there were around 700 WebKit vulnerabilities reported in that time frame.
    4. Fourth, the default browser in Android uses WebKit, yet they weren’t counted against that platform.
  5. The results include 16 vulnerabilities in Safari itself (or in WebKit and just not diagnosed as such), the default browser.
  6. At least 4 of the 238 are vulnerabilities in Google Chrome (as opposed to WebKit) with no mention of iOS in the CVE.
  7. A wide variety of iOS applications are included in the list including Office Viewer, iMessage, Mail, Broadcom BCM4325 and BCM4329 Wi-Fi chips, Calendar, FreeType, libxslt, and more.

When you factor in all of the above, Android likely comes out on top for the number of vulnerabilities when comparing the operating systems. Once again, vulnerability statistics seem simple on the surface. When you consider the above, and further consider that there are likely more points that influence vulnerability counts, we see that it is anything other than simple.

The curiously creeping value of the iOS vulnerability…

[This was originally published on the OSVDB blog.]

The market for vulnerabilities has grown rapidly the last five years. While the market is certainly not new, going back well over ten years, more organizations are interested in acquiring 0-day / private vulnerabilities for a variety of needs. These vulnerabilities cover the gambit in applications and impacts, and range from the tens of dollars to $100,000 or more. While such transactions are sometimes public, high-end vulnerabilities that sell for large sums generally are not a matter of public record. That makes it difficult to track actual sale prices to gauge the value of such vulnerabilities.

In the vulnerability market place, the seller has the power. If they hold a 0-day vulnerability that is in demand, they can set their own price. For the few vulnerability brokers out there, the perception of vulnerability value is critical for their business. In March, 2013, a Forbes piece by Andy Greenberg covered this topic and told of the sale of an iOS vulnerability that allegedly sold for $250,000.

Even with the $250,000 payout [the Grugq] elicited for that deal, he wonders if he could have gotten more. “I think I lowballed it,” he wrote to me at one point in the dealmaking process. “The client was too happy.”

As expected, there is no validation of the claim of the sale. The price tag comes from the vulnerability broker who has an interest in making such prices public, even if they are exaggerated. Jump to July, 2013, and a New York Times article by Nicole Perlroth and David Sanger makes a vague reference to an iOS vulnerability that sold for $500,000.

Apple still has no such program, but its vulnerabilities are some of the most coveted. In one case, a zero-day exploit in Apple’s iOS operating system sold for $500,000, according to two people briefed on the sale.

Given the vague details, it is fairly safe to assume that it references the iOS vulnerability sale from a year earlier. The NY Times article sources many people regarding vulnerability value, including thegrugq on the first page. This means the vague reference to the “two people briefed on the sale” were likely people briefed by thegrugq as well. Ultimately, this means that both articles and both figures, all source to the same person who has a decided interest in publishing high numbers. Without any detail, the journalists could have contacted one or both sources via email, meaning they could have just as well been thegrugq himself.

I find it interesting that in the span of 1 year and 4 months, the price of that iOS vulnerability jumped from $250,000 to $500,000. More to the point, the original $250,000 price is way out of the league of the prices of vulnerabilities at that time, on any market. Some of us were speculating that a (truly) remote vulnerability in a default Windows installation would go for around $100,000, maybe more. Even if you double our suspected price, it wouldn’t surprise me that a nation-state with a budget would purchase for that amount. But an iOS vulnerability, even remote without user interaction, a year ago? That doesn’t make sense given the user-base and distribution.

Even more interesting, consider that 4 days after the NYTimes article, another outlet was reporting the original $250,000 price.

As I mentioned before, none of this is close to being verified. The only source on record, is someone who directly benefits from the perception that the price of that vulnerability is exceedingly high. Creating the market place value of vulnerabilities through main-stream media is brilliant on his part, if what I suspect is true. Of course, it also speaks to the state of journalism that seemingly no one tried to verify this beyond word-of-mouth.

Are we so desperate?

The current state of U.S. politics is beyond dismal and entirely depressing. Our society only follows the corporate controlled ‘news’ channels that stump for their party-of-choice. Our congress is laughably ineffective in doing their job. The bi-partisan House/Senate can’t agree on anything, and little gets done in the pathetic number of days these overpaid politicians get paid to work.

Long ago, I told people that I would never vote for anyone that wants to be elected for a given office. If John Doe is running for President, I will not vote for him. He wants that position for a reason, and it isn’t his altruistic nature. The money, the power, and the fame drive people to politics. Even the no-name positions that we don’t care about, they are stepping stones for bigger offices.

The last few weeks remind me of why I will never vote for someone who wants office. They also remind me that our political system is full of manipulative unethical criminals, and little more.

Do you remember Eliot “Client 9” Spitzer, then Governor of New York was caught in a prostitution ring? As the Daily Show notes, he is now very forthcoming about his past. Of course, when he was caught soliciting prostitutes, that led him to now admit he didn’t maintain the ethical standards expected of him and led to his resignation.

Do you remember New York representative Anthony Weiner getting caught sending pictures of his penis to a female via Twitter? When challenged, he lied and claimed his account was ‘hacked’. Ultimately, because of his lies and unethical behavior, he resigned in the wake of the scandal.

Disagraced politicians, a dime a dozen these days. The number of anti-gay Republicans caught in homosexual encounters is enough to fill a blog. The amount of peace-loving anti-big-government Democrats that don’t support everything they claim to support could fill another blog. [Insert sentence about third political party going against their claims… oh wait.]

So jump forward to 2013. The U.S. population is around 313.9 million people. Are we so strapped for people that want to hold office, that we would actually consider someone who has already proven they do not have the ethical merit in recent years? Apparently, we are. Is it the fact that the oath many take simply says to uphold the Constitution, which most of them have a history of going against, and does not say to be an unethical lying slimeball jackass? Does that absolve them of their past transgressions, and make them justified and righteous in the public’s eye? Apparently, it does.

Eliot “Client 9” Spitzer is running for office again. Insult to injury? Spitzer has not “filed an ethics report on time with the city Conflicts of Interest Board.” If that wasn’t bad enough, and apparently it isn’t, Spitzer’s primary competitor is Kristin Davis. Who is that you ask? According to her, the leader of the escort service that supplied Spitzer with the prostitutes. Personally, I think she is telling the truth. After all, why would you lie about that going into politics?

Anthony “show my junk on Twitter” Weiner is also running for office again.

Remember Mark Sanford of South Carolina, who lied about his extramarital affair, claiming he was hiking in the mountains? Yeah, he is back, and running again too.

So again, I have to ask. Are we so desperate for politicians that we’d happily take these unethical assholes back and let them hold public office? Apparently so. Some of them are ahead in the polls already.

If you ever wonder why I don’t vote, remember this. If someone wants political office, I don’t want them. If someone has proven themselves unethical, I don’t want them. Until I see an election between two relatively honest and ethical people, then I refuse to vote. In doing so, I do not give up my right to complain about the election or outcome. Telling me I have to vote for one of two lying assholes to ‘gain’ the right to complain, means you don’t understand what our country was founded on.

At this point, I will happily accept the Harrison Bergeron model of politics.

eliot-spitzer

The Popcorn Thesis

During a recent email thread, a friend and I were comparing our local squirrels. She put forth that her Chicago squirrels did not eat popcorn, to which I expressed my disbelief. I couldn’t imagine a squirrel turning their nose up at it. I said I would have to test that theory.

I’ll be curious whether yours will eat it. Maybe the consistency of popcorn is too close to foam packaging peanuts, and my squirrels know not to eat foam peanuts based on prior unfortunate digestive disturbances? Dunno…

Maybe that hippie organic crap stuff you try to pass off as popcorn is the problem I say! Give them good old-fashioned regular popcorn and see what happens. I did! First, start with some plain old microwave popcorn:

20130713_222727

Ensure it is nuked properly, and not burned:

20130713_223147

Leave some out with the usual offering of almonds and townhouse mini-club crackers before bed:

20130713_223315

The verdict? My squirrels are not food snobs. All three foods demolished by 9AM. Victory!

20130714_103234

(I called this a Thesis because said friend is an academic, and figured it would resonate better.)

Exploding the Review

In the early 90’s, when I was moving in the world of computer bulletin board systems (BBS), it ultimately ended in my interest in phreaking. It started out reading t-files, moved into wardialing, and a few years later would result in PBX, voice mail, and switch hacking. While I got a late start in the phreaking world, it involved a world of reading including years of historical activity related to the phone system. Blue boxes were all but a thing of the past. Rumors of a switch or two still allowing you to seize a trunk floated around, but the time and effort of building a box based on rumor wasn’t so appealing, especially after some thirty years of it being the primary tool of the trade.

Red boxes still worked and were fun. Like the phreaks before me and my friends, we didn’t have many people to call, but it was fun using them. Something about that spoofed quarter signal, dee-dee-dee-dee-dee in rapid succession. From there it was the world of voice mail hacking. At first, just to see what the system were about. That quickly morphed into trying to find out which ones allowed outdial, putting me on the eternal hunt for diverters. At some point, enough information emerged about switches, and after a chance lesson from a veteran, a few of us learned the absolute basics of the 1AESS switch. Within a year or two, the Internet was taking a hold of our minds. Mind you, this was when DNS was still largely controlled via your own HOSTS.TXT file, before BIND was prevalent.

I offer this history because it heavily influences this review, and my enjoyment of a book.

Cover-New-200

Exploding the Phone gives a fairly comprehensive history of the origins of phreaking (phone system hacking). Written by Phil Lapsley, foreword by Steve Wozniak, the book was published earlier this year.

The book “Exploding the Phone” opens with a curious story of a classified advertisement in the Harvard Crimson student newspaper:

WANTED HARVARD MIT Fine Arts no. 13 notebook. (121 pages) & 40 page reply K.K. & C.R. plus 2,800; batter; m.f. El presidente no esta aqui asora, que lastima. B. David Box 11595 St. Louis, MO 63105.

This story is a launching point into the curious world of the early phone hackers, known as “phone freaks” that later became “phone phreaks”. After a brief history of the creation of the phone system, Lapsley takes us through the early world of blue boxing. By sharing the stories of several early phreaks that independently discovered the 2600hz signal and how it could give them free calls and the ability to explore the phone system, we see that an entire generation of what is now known as ‘hackers’ were in it for the love of system, nothing more. Because nothing can be that pure, we also learn of bookies in the 60’s that used phreak-made blue boxes for profit, by evading long distance bills for their numerous calls. Along with the phreaks are the stories of the phone company security and law enforcement that began to investigate them.

We get detailed stories of blind phreaks like Josef Engressia (aka The Whistler), Bill Acker, and Rick Plath. Instead of rumors and lore, Lapsley took extensive time not only researching them, but speaking with them when possible. The stories continue with the phone company struggling to figure out this new wave of people using the system in ways not intended. The reader enjoys some of the classic pranks pulled by phreaks, as they routed their calls all over the world, even to the Vatican. The history lesson continues with the tale of John Draper, aka Captain Crunch, who did not discover the cereal-box whistle blew the 2600hz tone (he was told that by phreaks that figured it out years before). As with all hacker culture, the drama of snitching and trying to evade serious punishment enters the picture. The book wraps up with more recognizable names like Steve Jobs and Steve Wozniak, and their founding of Apple based on selling blue boxes.

To anyone remotely interested in phreaking, or phone systems in general, I highly recommend this book. The author has done a wonderful job outlining the past through colorful stories, new details, and a great sense of what the culture was like.

T-Mobile’s Poor Implementation Works Against Amber Alerts

Just over a month ago, I received a pop-up alert on my Samsung Galaxy 3 (via T-Mobile) with a standard, and persistent, emergency broadcast noise…

Emergency alert
Longmont, CO AMBER Alert: LIC/245FLJ (CO) 2001 Blue Ford F350 Pickup truck
Type: AMBER Alert

The noise stopped briefly, then picked back up again until I tapped “OK”. This is a radical departure from the previous product behavior and service provided. Presumably this came with the latest Android update T-Mobile pushed shortly before (May 13).

No warning about this change, no indication where the alerts are coming from, no explanation on criteria for receiving (Longmont is almost 40 miles north of me, outside a metropolis of ~ 4.5 million), no indication of how often we receive them, a repeating noise that we have to acknowledge (as opposed to SMS that gives a noise/vibration one time only), etc. I’m not opposed to getting such warnings but I should be able to opt in and control the settings for how it is displayed.

One hour later, I received the same alert. That is intrusive and annoying. When it happened, I thought “if this shit happened at night, it would wake me up and force me to get up to ack the alert and turn off the phone” and just that happened. Wednesday early morning, at 5:20AM I received another. As I thought, it woke me, given the emergency sound and vibrating on my desk.

amber_alert

Looking at the SMS options that control this is also interesting. I now have to receive “Presidential Alerts” and cannot opt out of them. There are also Imminent Extreme alerts, Imminent Serious alerts, and the Amber alerts that I have received twice now. What are the others, and what differentiates them? When was the last time a Presidential broadcast was sent to everyone’s email address or home phone number? Absurd you say, why is it all of a sudden OK to send them to every subscriber’s cell phone?

What bothers me the most is that the Amber alerts, and presumably the others, do not adhere to the rest of my SMS settings. When I get an SMS, it vibrates once, makes an audible noise of my choice once, and sits idle until I check the phone. Amber alerts come up with a different sound; one that repeats until I acknowledge it.

Screenshot_2013-05-25-17-28-55

This is ridiculous. I want to receive them, but on my terms. The current setup and being woken at five in the morning forced me to disable the Amber alerts. T-mobile’s crappy technical implementation has worked contrary to their intentions by annoying customers into disabling them. This works against the entire purpose of having the alerts pushed via cell phones.