A critique of the summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”

What do you think of this?” It always starts out simple. A friend asked this question of an article titled Summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”. This study is math heavy and that is not my jam. But vulnerability databases are, and that includes the CVE ecosystem which encompasses NVD. I am also pretty familiar with limitations of the CVSS scoring system and colleagues at RBS have written extensively on them.

I really don’t have the time or desire to dig into this too heavily, but my response to the friend was “immediately problematic“. I’ll cliff notes some of the things that stand out to me, starting with the first graphic included which she specifically asked me about.

  • The header graphic displays the metrics for the CVSSv3 scoring system, but is just labeled “CVSS”. Not only is this sloppy, it belies an important point of this summary that the paper’s work is based on CVSSv2 scores, not CVSSv3. They even qualify that just below: “We should note the analysis conducted by Ross et al. is based upon the CVSS Version 2 scoring system…
  • Ross et al. note that many exploits exist without associated CVE-IDs. For example, only 9% of the Symantec data is associated with a CVE-ID. The authors offered additional caveats related to their probability calculation.” That sounds odd, but it is readily explained above when they summarize what that data is: “Symantec’s Threat Database (SYM): A database extracted from Symantec by Allodi and Massacci that contains references to over 1000 vulnerabilities.” First, that data set contains a lot more than vulnerabilities. Second, if Symantec is really sitting on over 900 vulnerabilities that don’t have a CVE ID, then as a CNA they should either assign them an ID or work with MITRE to get an ID assigned. Isn’t that the purpose of CVE?
  • Ross et al. use four datasets reporting data on vulnerabilities and CVSS scores…” and then we see one dataset is “Exploit Database (Exploit-DB): A robust database containing a large collection of vulnerabilities and their corresponding public exploit(s).” Sorry, EDB doesn’t assign CVSS scores so the only ones that would be present are ones given by the people disclosing the vulnerabilities via EDB, some of whom are notoriously unreliable. While EDB is valuable in the disclosure landscape, serving as a dataset of CVSS scores is not one of them.
  • About 2.7% of the CVE entries in the dataset have an associated exploit, regardless of the CVSS V2 score.” This single sentence is either very poorly written, or it is all the evidence you need that the authors of the paper simply don’t understand vulnerabilities and disclosures. With a simple search of VulnDB, I can tell you at least 55,280 vulnerabilities have a CVE and a public exploit. There were 147,490 live CVE IDs as of last night meaning that is almost 38% that have a public exploit. Not sure how they arrived at 2.7% but that number should have been immediately suspect.
  • In other words, less than half of the available CVSS V2 vector space had been explored despite thousands of entries…” Well sure, this statement doesn’t qualify one major reason for that. Enumerate all the possible CVSSv2 metric combinations and derive their scores, then look at which numbers don’t show up on that list. A score of 0.1 through 0.7 is not possible for example. Then weed out the combinations that are extremely unlikely to appear in the wild, which is most that have “Au:M” as an example, and it weeds out a lot of possible values.
  • Only 17 unique CVSS vectors described 80% of the NVD.” Congrats on figuring out a serious flaw in CVSSv2! Based on the 2.7% figure above, I would immediately question the 80% here too. That said, there is a serious weighting of scores primarily in web application vulnerabilities where e.g. an XSS, SQLi, RFI, LFI, and limited code execution could all overlap heavily.
  • Input: Vulnerabilities (e.g., NVD), exploit existence, (e.g., Exploit-DB), the number of clusters k” This is yet another point where they are introducing a dataset they don’t understand and make serious assumptions about. Just because something is posted to EDB does not mean it is a public exploit. Another quick search of VulnDB tells us there are at least 733 EDB entries that are actually not a vulnerability. This goes back to the reliability of the people submitting content to the site.
  • The authors note their approach outperforms CVSS scoring when compared to Exploit-DB.” What does this even mean? Exploit-DB does not do CVSS scoring! How can you compare their approach to a site that doesn’t do it in the first place?

Perhaps this summary is not well written and the paper actually has more merit? I doubt it, the summary seems like it is comprehensive and captures key points, but I don’t think the summary author works with this content either. Stats and math yes. Vulnerabilities no.

Not all CVEs are Created Equal. Or even valid…

[I wrote this early 2019 and it was scheduled for January 7 but it apparently did not actually publish and then got lost in my excessive drafts list. I touched it up this week to publish because the example that triggered this blog is old but the response is evergreen. Apologies for the long delay!]

I recently caught a Tweet from @NullCon offering 10 free conferences passes to NullconDasham, awarded to “InfoSec heroes who shared their hard work with the community & contributed to the @CVEnew database“.

I re-Tweeted with comment pointing out “There were at least 311 CVE assignments in 2018 alone, that were for issues that were *not* a vulnerability. I hope you are going to scrutinize the submissions.” Anant Shrivastava replied asking for some examples, and the next morning Mitja Kolsek suggested a blog would be beneficial. Here it is, as short and sweet as I can, which is never short in the world of Vulnerability Databases (VDBs) due to caveats.

Continue reading

More authorities, more CVEs; Oh, and more commentary.

On November 10, TechBeacon published a great article by Rob Lemos titled “More authorities, more CVEs: What it means for app sec teams” in which I was quoted, along with several other people.

Like many articles of this nature, those who provide input often will talk for as long as half an hour and ultimately get a couple lines quoted. We do it to provide background and context on the topic as well as have an open discussion on vulnerability trends. That means there are ‘outtake’ opinions and facts, as well as our potential reaction to other parts of the article that did not include our input. So this blog just covers some of my random observations to compliment the article.

Until 2016, more than 80% of software security issues assigned a CVE identifier belonged to only 10 classes, or weaknesses, as classified by their Common Weakness Enumeration (CWE) category. But in 2019, the top 10 weaknesses only accounted for 59% of reported vulnerabilities.

The Common Weakness Enumeration (CWE) index is interesting to me and I wonder if it has gotten so big to degrade its value. Consider that there are now 891 CWE identifiers as of August 20 in version 4.2 of the framework. Per the article, only 10 of them account for 59% of vulnerabilities which will no doubt include XSS, SQLi, and CSRF as examples. That makes me wonder the value of abstracting so much as it means that hundreds of those CWEs will represent a handful of vulnerabilities at most.

Digging into the 2,298 page PDF documenting version 4.2, you can jump toward the end of the CWE list and see that several have been created but have no “Observed Examples”. In fact, searching for that phrase only yields 397 hits. Does that mean that out of 891 CWE IDs representing weaknesses, that MITRE has only come up with 397 that match known vulnerabilities? I certainly expect otherwise and hope this is just documentation shortcoming as I feel that every CWE ID should be linked to a concrete real-world example. I’d love to see

I’d love to see a simple breakdown of the top 100 CWE along with how many vulnerabilities are associated with them (via NVD, since MITRE doesn’t actually apply CWE to entries) and what percentage of the overall vulnerabilities that represents. It might be very telling just how useful CWE is and if the project is being pushed too heavily from an academic standpoint. Before you judge that comment, let me know how useful this CWE report from MITRE is, and make sure you load it in Chrome.


It’s an open question whether the addition of coding repositories will lead to another expansion in the number of vulnerabilities.

I don’t think that is an open question at all. I think the number of vulnerabilities will go up as a result of more coding repositories becoming a CNA. But that isn’t really the issue here. Instead, the real questions should be centered around what quality of CVE entries they will provide, if they will adhere to CNA standards, and if MITRE will enforce CNA policy on them.

Based on history, the short answers to those questions are: quality will go down, no, and no. As soon as MITRE provides a breakdown of how many IDs were published by each CNA it is difficult to know. Speaking of, why hasn’t MITRE published such statistics? Rhetorical question, apologies.


Open source vulnerabilities: Do you know what’s in your software?

I, along with many others, can’t stress this enough! Please make sure you understand what third-party software your developers are using. This affects your organizations from both a vulnerability standpoint, as well as legal accountability. Using a third-party library against its license could open you up to some hardships.

The only reason I quoted this section is because I just read an article in the latest Wired that mentions Bootstrap is thought to be used on nearly 20% of all web sites across the Internet. That is incredible.


Patch Tuesday is becoming a bottleneck

There is a lot more that can be said on this topic. It reminds me of a 2015 blog I wrote that actually goes back farther to 2007 where this problem was predicted long before the nightmare IT teams experience once a month. It’s only going to get worse as more vendors jump on this patch schedule and the only ones who will suffer are their paying customers. But hey, let’s let them keep using the term “responsible” disclosure too.


But exploitability alone doesn’t solve the problem—three quarters of the 17,300 vulnerabilities identified ranked CVSS exploitability rating of 8.0 or higher.

I’m not sure a more perfect example of why CVSS has become worthless exists. On its own, especially using the Base score only, is it really helpful that so many vulnerabilities are ‘High Risk‘? This is also a good reminder of another blog I have been meaning to write for a while that outlines the distribution of CVSSv2 versus CVSSv3 and how it impacts scoring. With a couple charts you will get a nice visual of just how poorly thought out some of the framework was. Of course, this has already been evaluated by others years back as well.


Finally, because I don’t hold the copyright to the picture used in the TechBeacon article header, I offer my version:

Picture of me by D2d in November, 2018 at Tomayo, in Denver, CO.

“The History of CVE” and A Couple of Objections

I just read “The History of Common Vulnerabilities and Exposures (CVE)” by Ary Widdes from Tripwire and found it to be a great summary of the 20+ years of the program. I say that as an outspoken CVE and MITRE critic even! I do have a couple of objections however, with the conclusion, and then a fun bounty!

Widdes concludes the history by saying:

A lot has changed in the 21 years since the CVE List’s inception – both in terms of technology and vulnerabilities. Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage. It’s also possible that someone else would have created a service similar to the CVE List. Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

There’s a lot to unpack here so I will take it one sentence at a time, starting with the second.

“Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage.”

No, there is no “possible” here. That is a simple reality with an important caveat. The reality is that teams of all types still use multiple tools from multiple vendors to do their job. The caveat, and more to the point of that sentence, is that CVE doesn’t offer “complete coverage” and many of the vulnerability scanners only cover a third of the issues in CVE for various reasons. Even using a combination of firewalls, vulnerability scanners, intrusion detection/prevention, audits, and a slew of other tools, organizations are likely seeing half of what CVE has to offer at best. Widdes’ conclusion here gives undue credit to CVE and the state of vulnerability coverage it offers.

It’s also possible that someone else would have created a service similar to the CVE List.

This is where the vulnerability historian in me wants to rage a bit. This statement is unequivocally false for the simple reason that vulnerability databases existed before CVE, both free (e.g. X-Force) and commercial (e.g. RSI), in 1997 alone [1]. The first vulnerability database was created in 1973, specific to Multics, but also when there weren’t that many other systems to catalog bugs or vulnerabilities in. In 1983 we saw the Mt Xinu Bug List and in 1985 Matt Bishop’s List of UNIX Holes, both of which were more comprehensive than one platform. If we consider a vulnerability database implemented via product, we had ISS, SATAN, Ballista, and Nessus between 1995 and the creation of CVE in 1999. Many of the hackers turned security professionals may fondly remember Fyodor’s Exploit World (1996 – 1998) from both aspects of their lives. Those same folks probably also remember Packet Storm (1998) which is still running today.

Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

This, unfortunately, is true. I say unfortunately because of my long-standing criticisms of CVE over the past decade, but won’t go into here.

Bug(s) Bounty:

If there is anyone at MITRE open to outright bribery, including all-you-can-eat sushi dinners, I will pay a bounty to get my hands on that list of 8,400 submissions! While I know there are likely a lot of duplicates, the vulnerability historian in me would love to audit that data to see if MITRE decided to skip any that would be considered vulnerabilities by today’s standards, or where someone else back then had more knowledge of a vulnerability than was submitted. That data is over twenty years old and was solicited, processed, and partially published with U.S. taxpayer funded money. There’s no reason not to make it public. =)

[1] The Repent Security Inc. (RSI) database existed in 1997 but may not have been offered as a commercial product until 1998.

Why @anacondainc Doesn’t Fully Understand CVEs

It’s worrisome that in 2020 we still have people in influential technical roles that don’t understand CVE. A friend told me earlier this year he was in a meeting where someone said that CVE IDs are assigned in order, so CVE-2020-9500 meant there were 9500 vulns in 2020 so far. Of course that is not how it works and a dangerous understanding of CVE.

I ran across an article written by Nick Malkiewicz of Anaconda titled “Why Understanding CVEs Is Critical for Data Scientists“. This article has several bits that show a lack of understanding of what CVE is. One of the biggest is equivocating a CVE with a vulnerability. Yes, many vulnerabilities directly map to a single CVE identifier, but a CVE is the identifier not the vulnerability. Additionally, sometimes one vulnerability can track with multiple CVE IDs, or one CVE ID can track to multiple vulnerabilities. So lines like the following are concerning:

When someone finds a CVE, they report it to a CVE Numbering Authority (CNA).

When someone finds a vulnerability, they report it to MITRE or a vendor, who may be a CNA but more often not one. That vendor can then ask MITRE for an ID via a web form.

CNAs assign identification numbers to CVEs and list them in publicly accessible databases.

A CNA is required to inform MITRE after a CVE-assigned vulnerability has been disclosed. That is actually a fairly recent rule, implemented in the last few years. For most of CVE’s history there was no requirement or specific communication channel for a CNA to notify MITRE of this. That was one of many failings of the CVE ecosystem and directly led to companies being breached, as they relied on CVE to be ‘complete’ and timely.

Each vulnerability listed in a CVE database has a score from .1 to 10, 10 being the highest risk level. These scores are based on exploitability, impact, remediation level, report confidence, and other qualities.

Technically, not even the first line is true as NVD can score a vulnerability as 0.0, meaning it is not a vulnerability and poses no risk. This occurs when a researcher or vendor disclose a vulnerability but don’t fully understand the issue or the subsequent impact. This happens hundreds of times a year although many are not included in NVD. The second sentence from Anaconda is also incorrect as NVD only scores CVSS Base scores. The exploitability, remediation level, and report confidence are part of Temporal scores and not included. You can see an example with CVE-2020-2800 published by Oracle and given a CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N score by both Oracle and NVD. This misunderstanding of NVD CVSS scoring is more baffling as Anaconda links to the same FIRST CVSS document I do in this paragraph.

Anaconda goes on talking about how there are other factors at play including looking at the history of a package, how fast vendors respond, and more. This is great advice and critical for companies facing tens of thousands of vulnerabilities a year. Unfortunately, they slide into the “more lipstick on that pig” problem:

The good news is, there are tools that automate the CVE monitoring process.

This is true. But, more ways to manipulate bad data still leaves you with bad data. In addition to CVE missing several thousand vulnerabilities a year, their push for quantity in the last few years have led to a serious hit on quality. There are some CVE IDs that have descriptions missing critical information like the vendor, affected version, or impact. All the data wrangling and fancy analysis of that data is still based on bad or incomplete information. All the lipstick on that pig still makes it a pig.

Finally, I will quote on other line from their blog that is curious:

Hacking open-source software also has a bigger payoff because many more people use it.

I understand and appreciate the sentiment, and finding a vulnerability in a library like OpenSSL obviously has a huge impact. However, that and a couple dozen libraries are still the outliers in the bigger picture. Compare your vulnerabilities like EternalBlue to a standard open source library vulnerability and they are not even close as far as how “many more people use it”.

Disclosure Repair Timelines?

For those in InfoSec, you have probably seen a vulnerability disclosure timeline. Part of that often includes the researcher’s interaction with the vendor including the vulnerability being fixed. After the issue is disclosed, the story typically ends there. Every so often, work needs to be done after that to ‘repair’ part of the disclosure.

For the last year or more I have found myself having to follow-up on more disclosures, specifically because someone on Twitter has posted using an incorrect CVE ID associated with the vulnerability. One of the cornerstones of a CVE assignment is to give it a unique identifier that makes that vulnerability distinct from any others that may be similar. Using the incorrect CVE ID can actually cause a lot of headache for threat intelligence folks that monitor for vulnerability disclosures.

Often times I send a message and within a day the errant CVE ID is fixed. The errors tend to be nothing more than a typo or transposition issue. When fixed quickly and not further indexed by search engines and cited or included by news aggregator sites, the problem is over. Once the errant ID is in several reputable (or somewhat reputable) sources, it is more prone to be quoted in additional blogs and spread from there. Catching and fixing these errors needs to happen quickly, but unfortunately MITRE, the organization responsible for CVE, does nothing in this regards.

The past two weeks, I ran into what is probably the worst case as far as time and effort required to fix a single incorrect CVE. I thought I would share what the timeline looks like as this is not something anyone typically tracks that I am aware of, myself included. But it shows that even after a disclosure more work may need to be done to ensure clarity in it. I’m withholding names because while this time around was difficult, the journalist and publication has quickly fixed other typos in the past. My goal is to show that timely corrections are what is best for the community.

4/30 – Article published citing four CVE IDs, one incorrect.
4/30 – Ping publication/journalist on Twitter
5/1 – Bump thread
5/8 – Bump thread
5/13 – Tweet again asking for a correction
5/14 – Submit site feedback via two different forms
5/14 – Tweet frustration at publication
5/15 – Publication replied to form, didn’t seem to fully understand the point
5/19 – Sent a DM to the author of the article pointing to original Tweet
5/21 – Author replied saying they will fix it. Article amended to fix and clarify the error.

Twenty one days to fix is rough. Publications and journalists; please understand that a CVE ID is important to get right. If you have any questions about CVE, how it works, or the importance, please feel free to reach out. I am happy to take the time to help you.

WhiteSource on ‘Open Source Vulnerability Databases’ – Errata

[This was originally published on the OSVDB blog.]

On September 8, 2016, Jason Levy of WhiteSource Software published a blog titled “Open Source Vulnerability Database”. Almost two years later it came across my radar and I asked via Twitter if WhiteSource was interested in getting feedback on the blog, since it contained errata. They never replied and it fell off my radar. Last week, someone asked me about WhiteSource’s claims about maintaining the largest vulnerability database. While I have not seen it, I expressed doubt and dug into my notes. Until then, I had forgotten that I created notes to write a blog listing the errata. Since their name came up again, I figured it is worth revisiting this, especially since they never replied to me.

First, I was a long-term contributor to OSVDB, an officer in OSF (the 501(c)(3) that managed the project), a founder at Risk Based Security, and one of the maintainers of VulnDB since day one. Don’t worry, all of that is explained in more detail below. Here are quotes from their blog and my thoughts on the errata:

The open source community has been living up to this statement recently, with the accelerated rate of discoveries of open source vulnerabilities reported by such sources as the NVD open source vulnerability database.

CVE/NVD have dismal coverage of open source libraries in the big picture. Their coverage is primarily through a couple dozen projects that seek out CVE IDs, researchers that request CVE IDs, and the OSS-Sec mail list. To date, there are still many developers, projects, and even security researchers that don’t know what CVE is.

The NVD is by far the main source for researching vulnerabilities.

This sentence does not make sense. As you note below, NVD does not research vulnerabilities; they don’t even aggregate vulnerabilities themselves. While many consume NVD’s data over CVE, due to the format they make the data available, more “analysis” is done via CVEDetails.com than NVD in my experience.

Their analysis contains information like how the vulnerability operates, its impact rating, CVSS score, and links to any available patches/fixes.

The main purpose of NVD, aside from a burden on taxpayer’s dollars, is to provide CPE and CVSS scores for vulnerabilities via CVE. They rarely, if ever, do additional analysis of the vulnerability unless required to determine a CVSS score. If they do, they do not “show their work” so to speak, instead using the same CVE description. As best I am aware, they don’t dig up additional references either, so the only solution links provided are via CVE. That said, I haven’t audited NVD in several years to see if that still holds true, but I would be surprised if anything has changed.

The OSVDB (open source vulnerability database) was launched in 2004 by Jake Kouhns, the founder and current CISO of Risk Based Security – the company which now operates OSVDB’s commercial version, the VulnDB.

Jake Kouns, one of the founders of Risk Based Security (RBS) did not launch OSVDB. And that is the Open Sourced Vulnerability Database. OSVDB was created and launched by H.D. Moore. More history is available via Wikipedia. While OSVDB was a basis for the historical data in VulnDB, Risk Based Security funded OSVDB entirely for over two years and it was their data that was shared publicly via OSVDB before the project shut down. When OSVDB shut down, RBS continued to maintain VulnDB which contains over eight years of vulnerability information not found in OSVDB. With considerable resources and a team maintaining VulnDB, it has become something OSVDB wished it could, but never came close.

This commercial version continues to be maintained by Risk Based Security, but without the strong community that maintained the OSVDB.

There was no “strong community” that maintained OSVDB. For years, it limped along on the backs of a handful of volunteers, with very little support from the community; financial or labor. At times, there were only two of us maintaining the project while most volunteers signed up, worked on the database for 20 minutes, and never returned to help again.

The OSVDB/VulnDB is perceived by many in the community as redundant, for the majority of vulnerabilities are also reported in the NVD.

This is WhiteSource’s ignorance or bias at play. OSVDB was known for having tens of thousands of vulnerabilities not found in CVE/NVD. One of VulnDBs strong points is that it contains almost 69,000 vulnerabilities not found in CVE/NVD. Since WhiteSource claims to have a comprehensive database, you should have disclaimed this while spreading false information about VulnDB. Since VulnDB is not public, the notion that is perceived by “many” in such a light is just your marketing. Anyone reading VulnDBs quarterly or year-end reports can see the incredible gap between them and CVE/NVD. Also curious is that the link off “monitoring of multiple vulnerability databases including the CVE/NVD” just links to your blog on the topic. Other than CVE/NVD, which vulnerability databases do you monitor exactly?

I’d also happily sign a non-disclosure agreement to do a quick audit of your database, to verify or refute your claim that you have the “the largest coverage of vulnerability listings“. I’d only ask that in return, I can publish a simple ‘yes’ or ‘no’ answer to that claim.

Security advisories are usually the first place that security professionals and developers look when they have security issues within a specific scope.

If this is WhiteSource’s methodology for maintaining a vulnerability database, it does not bode well for your customers. Formal security advisories are quickly becoming, or have already become, a minority in the disclosure game. OSVDB and VulnDB didn’t stay far ahead of the curve using this antiquated mindset. We knew to look elsewhere all along.

WhiteSource tracks almost a dozen security advisories (like RubyOnRails, RubySec and Node Security), to ensure our customers are alerted on new vulnerabilities the minute they’re reported.

A whole dozen? The VulnDB team currently monitors 3,100 sources every week. Some are monitored ~ hourly, some a few times a day, some once a week depending on various criteria such as publication frequency. If WhiteSource is only looking at those sources, and that type of source, it really doesn’t bode well for your claims of the largest vulnerability listing.

The majority of vulnerabilities are published with a remediation suggestion, like a new patch, new version, system configuration suggestion etc.

While technically true, per VulnDB data, it is better to qualify that. For example, in Q1 2019, 60.8% of disclosed vulnerabilities they aggregated had a documented solution. But, a couple caveats. First, that was down 13.5% over Q1 2018! Second, not all solutions are created equal. Just because there is a patch committed against the ‘master’ branch doesn’t mean an organization can actually use it. Their policy or available resources may not allow for one-off code patching like that.

And here at WhiteSource, this is exactly what we do.

Unfortunately, based on the way you describe what you do, you are not much better than CVE/NVD I’m afraid.

CVE and the matter of “unique” ID numbers

Common Vulnerability Enumeration, now known as Common Vulnerabilities and Exposures (CVE) is a vulnerability database (ignore their silly claim to be a ‘dictionary’) that the information security industry relies on heavily, unfortunately. Per MITRE’s CVE page, “CVE® is a list of entries—each containing an identification number, a description, and at least one public reference—for publicly known cybersecurity vulnerabilities.” Great, digging a bit deeper into their ‘About‘ page, we get a better definition:

Use of CVE Entries, which are assigned by CVE Numbering Authorities (CNAs) from around the world, ensures confidence among parties when used to discuss or share information about a unique software or firmware vulnerability, provides a baseline for tool evaluation, and enables data exchange for cybersecurity automation.

Please take note that a CVE Entry, or ID number, “ensures confidence” when discussing or sharing information about a vulnerability. Basically, it is supposed to be a unique ID to ensure that confidence. Despite that, any of my dozen loyal Twitter followers will see me constantly pinging researchers, vendors, and the media pointing out that they are using the wrong CVE number to reference a vulnerability. Often times it is a case of not copying and pasting, rather typing it out manually. It is also why in the vulnerability database (VDB) world, we strongly emphasize that copy/paste is the best thing to do in order to prevent transcription errors on an ID that is supposed to be unique.

Sure, it seems pedantic to a degree, but imagine if your doctor decided to manually transcribe a diagnosis code after your visit and you get a call saying you were diagnosed with something completely different. In the vulnerability world, it means you might be vulnerable to something and have no idea if so. If you are, you aren’t sure if there is a solution. Maybe a bit of a dramatic analogy? But.. it holds water, has a bit of recent history, and is the kind that helps administrators better understand the underlying issue.

Instead of sending out a series of Tweets on the latest example, I decided to write a blog to show how these little typos can snowball quickly. Any mature VDB will have a variety of processes to catch wind of a CVE ID that they haven’t seen before. It can be as simple as a live search on Twitter for ‘CVE’ (super noisy) or more technical means. If you run across an unknown CVE you Google it to start, that simple. Today’s example was CVE-2019-0895, which appeared to be a “new windows zero-day”. Exciting in the world of VDBs!

Let me go ahead and spoil things, to make this easier. These articles call it “2019-0895”, but in reality, they actually mean “2019-0859”. A simple transposition of numbers, which is all too common in prior cases. Based on ten-second review, it appears that Fossbytes was the first to transcribe these numbers (Unverified @fossbytes14 on Twitter?). A day later, extremely similar articles appeared on Prodefense (no Twitter and broken Contact form?) and “In Depth IT News / SecNews” which has some serious rendering issues in Chrome. The day after that, Tech Rights references it via a weird embedded link below in an unrelated article [1], and Tux Machines posted about it with key quotes cribbed from other articles, the Fossbytes article in this case.

In each case, it is clear that the offending typo came from Fossbytes. The “In Depth IT News” site even links to https://securelist.com/new-win32k-zero-day-cve-2019-0859/90435/ which has the correct CVE ID in the URL. It is absolutely clear that most of these sites are using automated crap to aggregate content and have no real desire to share accurate news. Each one of them are evidence to the damage caused by a single transposition error from Fossbytes, a “leading source of technology news with a focus on Linux distro releases” … that decided it was important to write about this critical Windows zero day? A critical zero day that is actually ten days old at the time of their article.

OK, hopefully we’re all on the same page here. My Twitter feed is a small graveyard of similar examples from the past few years. Each and every time, the “news” organizations that spread these bad IDs and introduce confusion and questions into the equation, and are the antithesis of a “news” site. Finally, I would like to go on the record about one more bit regarding CVE, which will come as no surprise. On the CVE ‘About’ page, it says CVE is:

Industry-endorsed via the CVE Numbering Authorities, CVE Board, and numerous products and services that include CVE

As a former ten-year veteran of the CVE Board, I do not endorse CVE.


[0] Note: If any of my links show a fixed version of the CVE, good! You can see the originals on archive.today.
[1] This should really be a separate blog post, but it would mostly be cursing around a simple concept; this is the problem with content/link aggregation sites… which are a plague on the Internet. In 2019, they aren’t trying to help, they are desperate attempts to make a few bucks. Disagree? They would have caught this error when they did a quick tech edit pass on the article. But they didn’t, because it is all automated and centered around ‘SEO’ (search engine optimization) so it appears in Google results and you click and see the ads they are serving. I bet if anyone dug deep on such sites, the amount of questionable traffic or malware they delivered might be enlightening. Go back to where this is linked from and notice the URL of the article (/2019/04/18/libreoffice-6-2-3/) and how far you have to scroll to get to the bottom of the page, past all the “content”.

Microsoft, CVE, MITRE, ETERNALBLUE, Headache…

2019-02-14 Update: Thanks to Chris Mills @ MSRC (@TheChrisAM), who has been working behind the scenes since this blog was published, he has brought clarity to these assignments! MSRC is still potentially touching up some additional documentation to make it easier to see these associations, but here is the definitive answer from him:

CVE-2017-0143 ShadowBrokers : EternalSynergy (Blog)
CVE-2017-0145 ShadowBrokers : EternalRomance (Blog)
CVE-2017-0144 ShadowBrokers : EternalBlue (Blog)
CVE-2017-0146 ShadowBrokers : EternalChampion (Blog)

Note that only the EternalChampion blog does not reference the associated CVE, but he is working on getting that updated. I have also recommended that MSRC update MS17-010 to use the codenames in that advisory as well. Apparently editing the actual bulletins takes a bit more work, but he’s on it! I can’t thank Chris enough for running with this and helping bring clarity to these assignments.


There was initially a lot of confusion over the Equation Group disclosure. Which were legitimate vulnerabilities, which were new, which were known, which were patched, and ultimately how they would be referred to other than their leaked nicknames. That is the purpose of The Common Vulnerabilities and Exposures project (originally Common Vulnerability Enumeration), to give a unique ID to a specific issue so that you can reference a vulnerability without question. A year and a half later? We’re still wondering apparently.

I contacted Microsoft Security Response Center (MSRC) on August 6, 2017 asking for clarification on the CVE assignment for one of the Equation Group vulnerabilities codenamed ETERNALBLUE, because their own resources contradicted each other. From my mail:

Per an older blog [1], the vulnerability known as ‘EternalBlue’ is assigned CVE-2017-0145. From the blog:

However, in this unique case, the ransomware perpetrators used
publicly available exploit code for the patched SMB “EternalBlue”
vulnerability, CVE-2017-0145, which can be triggered by sending a
specially crafted packet to a targeted SMBv1 server.

A newer blog [2] now lists it as CVE-2017-0144, which I believe to be incorrect. From the blog:

The new ransomware can also spread using an exploit for the Server
Message Block (SMB) vulnerability CVE-2017-0144 (also known as
EternalBlue), which was fixed in security update MS17-010 and was
also exploited by WannaCrypt to spread to out-of-date machines.

Can you confirm the correct assignment for ‘EternanBlue’ [sic], and due to the second blog, the assignment for ‘EternalRomance’, and update your blog(s) accordingly?

All this time later? MSRC never answered my mail, and never fixed one of the two blogs. CVE’s description of each does not mention the nickname in either entry. So the assigning CVE Numbering Authority (Microsoft), or CNA, and the core CVE project (MITRE) still don’t answer this question. To date, the Microsoft advisories for those two CVE ID still don’t mention the nickname. To add more confusion? Try using Google to find it, and you get a third CVE ID it may be (screenshot below). Although, that one result doesn’t actually have ‘EternalBlue’ in it, making us wonder why it is the sole result. The blog that MSRC originally published to add some clarity to the Equation Group still only references MS17-010 (and a dead link now). Looking at the new location for MS17-010 doesn’t find the nickname in the advisory either.

To this day, I am still fairly sure ETERNABLUE is CVE-2017-0145 and attribute it as such, but it sure would be nice if MSRC would clean up and clarify this mess.

Further, I have had to chase down two more errant CVE assignments by MSRC in the last months, which was fairly painful. After getting the runaround on both, being told to go ask Microsoft Support via a forum (despite MSRC being the definitive source for this information), not getting a reply, opening a new ticket with MSRC, reminding them that I was still waiting… those two finally got resolved after a month or more. I really don’t like casting shade on MSRC as over the years, in total, they have been wonderful to deal with. However, the last couple of years have seen a serious decline in this type of incident which should be ‘Vulnerability 101’, and a serious uptick in their resistance to clarify assignments when asked. Finally, if you are wondering why MITRE doesn’t provide some kind of oversight to this? Well they basically never have despite repeated requests for just that. Their only oversight is a ‘CNA Report Card’ that is more about statistics of assignments and such, and does not deal with the quality of assignments, incidents of confusion like this, or anything else that would be helpful to the community.

The only upside to all of this? I got to [sic] my own typo from the quoted email.

The Duality of Expertise: Microsoft

[This was originally published on the OSVDB blog.]

The notion of expertise in any field is fascinating. It crosses so many aspects of humans and our perception. For example, two people in the same discipline, each with the highest honors academic can grant, can still have very different expertise within that field. Society and science have advanced so we don’t have just have “science” experts and medical doctors can specialize to extreme degrees. Within Information Security, we see the same where there are experts in penetration testing, malware analysis, reverse engineering, security administration, and more.

In the context of a software company, especially one that does not specifically specialize in security (and is trivial to argue was late to the security game), you cannot shoehorn them into any specific discipline or expertise. We can all absolutely agree there is an absolute incredible level of expertise across a variety of disciplines within Microsoft. So when Microsoft releases yet another report that speaks to vulnerability disclosures, the only thing I can think of is duality. Especially in the context of a report that puts forth some expertise that they are uniquely qualified to speak on, while mixed with a topic that pre-dates Microsoft and they certainly aren’t qualified to speak on to some degree.

A Tweet from Carsten Eiram pointed me to the latest report, and brought up the obvious fact that it seemed to be way off when it comes to vulnerability disclosures.

carsten-ms-sir

The “MS SIR” he refers to is the Microsoft Security Intelligence Report, Volume 21 which covers “January through June, 2016” (direct PDF download).

It’s always amusing to me that you get legal disclaimers in such analysis papers before you even get a summary of the paper:

ms-disclaimer

Basically, the take away is that they don’t stand behind their data. Honestly, the fact I am blogging about this suggests that is a good move and that they should not. The next thing that is fascinating is that it was written by 33 authors and 14 contributors. Since you don’t know which of them worked on the vulnerability section, it means we get to hold them all accountable. Either they can break it down by author and section, or they all signed off on the entire paper. Sorry, the joys of academic papers!

After the legal disclaimers, then you start to get the analysis disclaimers, which are more telling to me. Companies love to blindly throw legal disclaimers on anything and everything (e.g. I bet you still get legal disclaimers in the footer of emails you receive, that have no merit). When they start to explain their results via disclaimers while not actually including their methodology, anyone reading the report should be concerned. From their “About this report” section:

This volume of the Microsoft Security Intelligence Report focuses on the first and second quarters of 2016, with trend data for the last several quarters presented on a quarterly basis. Because vulnerability disclosures can be highly inconsistent from quarter to quarter and often occur disproportionately at certain times of the year, statistics about vulnerability disclosures are presented on a half-yearly basis.

This is a fairly specific statement that speaks as if it is fact that vulnerability trends vary by quarter (they do!), but potentially ignores the fact that they can also vary by half-year or year. We have seen that impact not only a year, but the comparison to every year prior (e.g. Will Dormann in 2014 and his Tapioca project). Arbitrarily saying that it is a ‘quarter’ or ‘half-year’ does not demonstrate experience in aggregating vulnerabilities, instead it is a rather arbitrary and short time-frame. Focusing on a quarter can easily ignore some of the biases that impact vulnerability aggregation as outlined by Steve Christey and my talk titled “Buying Into the Bias: Why Vulnerability Statistics Suck” (PPT).

Jumping down to the “Ten years of exploits: A long-term study of exploitation of vulnerabilities in Microsoft software” section, Microsoft states:

However, despite the increasing number of disclosures, the number of remote code execution (RCE) and elevation of privilege (EOP) vulnerabilities in Microsoft software has declined
significantly.

Doing a title search of Risk Based Security’s VulnDB for “microsoft windows local privilege escalation” tells a potentially different story. While 2015 is technically lower than 2011 and 2013, it is significantly higher than 2012 and 2014. I can’t say for certain why these dips occur, but they are very interesting.

5year-win-lpe

Thousands of vulnerabilities are publicly disclosed across the industry every year. The 4,512 vulnerabilities disclosed during the second half of 2014 (2H14) is the largest
number of vulnerabilities disclosed in any half-year period since the Common Vulnerabilities and Exposures system was launched in 1999.

This quote from the report explicitly shows serious bias in their source data, and further shows that they do not consider their wording. This would be a bit more accurate saying “The 4,512 vulnerabilities aggregated by MITRE during the second half of 2014…” The simple fact is, a lot more than 4,512 vulnerabilities were disclosed during that time. VulnDB shows that they aggregated 8,836 vulnerabilities in that same period, but less than the 9,016 vulnerabilities aggregated in the second half of 2015. Microsoft also doesn’t disclaim that the second half of 2014 is when the aforementioned Will Dormann released the results of his ‘Tapioca’ project totaling over 20,500 vulnerabilities, only 1,384 of which received CVE IDs. Why? Because CVE basically said “it isn’t worth it”, and they weren’t the only vulnerability database to do so. With all of this in mind, Microsoft’s comment about the second half of 2014 becomes a lot more complicated.

The information in this section is compiled from vulnerability disclosure data that is published in the National Vulnerability Database (NVD), the US government’s repository of standards-based vulnerability management data at nvd.nist.gov. The NVD represents all disclosures that have a published CVE (Common Vulnerabilities and Exposures) identifier.

This is a curious statement, since CVE is run by MITRE under a contract from the Department of Homeland Security (DHS), making it a “US government repository” too. More importantly, NVD is essentially a clone of CVE that just wraps additional meta-data around each entry (e.g. CPE, CWE, and CVSS scoring). This also reminds us that they opted to use a limited data set, one that is well known in the Information Security field as being woefully incomplete. So even a company as large as Microsoft, with expected expertise in vulnerabilities, opts to use a sub-par data set which drastically influences statistics.

Figure 23. Remote code executable (RCE) and elevation of privilege (EOP) vulnerability disclosures in Microsoft software known to be exploited before the corresponding security update release or within 30 days afterward, 2006–2015

The explanation for Figure 23 is problematic in several ways. Does it cherry pick RCE and EOP while ignoring context-dependent (aka user-assisted) issues? Or does this represent all Microsoft vulnerabilities? This is important to ask as most web browser exploits are considered to be context-dependent and coveted by the bad guys. This could be Microsoft conveniently leaving out a subset of vulnerabilities that would make the stats look worse. Next, looking at 2015 as an example from their chart, they say 18 vulnerabilities were exploited and 397 were not. Of the 560 Microsoft vulnerabilities aggregated by VulnDB in 2015, 48 have a known public exploit. Rather than check each one to determine the time from disclosure to exploit publication, I’ll ask a more important question. What is the provenance of Microsoft’s exploitation data? That isn’t something CVE or NVD track.

Figure 25 illustrates the number of vulnerability disclosures across the software industry for each half-year period since 2H13

Once again, Microsoft fails to use the correct wording. This is not the number of vulnerability disclosures, this is the number of disclosures aggregated by MITRE/CVE. Here is their chart from the report:

ms-2016-1h16-chart

Under the chart they claim:

Vulnerability disclosures across the industry decreased 9.8 percent between 2H15 and 1H16, to just above 3,000.

As mentioned earlier, since Microsoft is using a sub-par data set, I feel it is important to see what this chart would look like using more complete data. More importantly, watch how it invalidates their claim about an industry decrease of 9.8 percent between 2H15 and 1H16, since RBS shows the drop is closer to 18%.

vulndb-vs-nvd

I have blogged about vulnerability statistics, focusing on these types of reports, for quite a while now. And yet, every year we see the exact same mistakes made by just about every company publishing statistics on vulnerabilities. Remember, unless they are aggregating vulnerabilities every day, they are losing a serious understanding of the data they work with.

March 19, 2017 Update – Carsten Eiram (@carsteneiram) pointed out that the pattern of local privilege escalation numbers actually follow an expected pattern with knowledge of researcher activity and trends:

In 2011, Tarjei Mandt while he was at Norman found a metric ton of LPEs in win32k.sys as part of a project.

In 2013, it was Mateusz Jurczyk’s turn to also hit win32k.sys by focusing on a bug-class he dubbed “double-fetch” (he’s currently starting that project up again to see if he with tweaks can find more vulns).

And in 2015, Nils Sommer (reported via Google P0) was hitting win32k.sys again along with a few other drivers and churned out a respectable amount of LPE vulnerabilities.

So 2012 and 2014 represent “standard” years while 2011, 2013, and 2015 had specific high-profile researchers focus on Windows LPE flaws via various fuzzing projects.

So the explanation is the same explanation we almost always see: vulnerability disclosures and statistics are so incredibly researcher driven based on which product / component / vulnerability type a researcher or group of researchers decides to focus on.