An 83 Word Excuse Instead of a 1 Character Fix (NCSC.nl)

The National Cyber Security Center of the Netherlands (NCSC.nl) has a curious take on sharing security information. On October 25, 2021 I contacted them to inform them of a simple typo in one of their advisories. I send mails or Tweets like this several times a week to researchers, vendors, and news outlets as CVE typos are prevalent. The issue is that a mistyped CVE ID can cause a lot of headache for organizations that take vulnerability intelligence seriously. Imagine an alert about a new CVE affecting enterprise software running your most critical assets and you can only find a news article on it saying the issue is critical. The vendor advisory doesn’t reference it and almost nothing to be found on Google or social media. What do you do? Before you spin up the entire team and tell them to stay late planning for emergency remediation, you need to know what you are dealing with.

Most of the time, the Tweets and emails get a quick reply acknowledging it was a typo and they fix it when possible. Every so often I get no reply and the typo stays there, likely forever. That typically happens on sites that appear to be automated aggregation of content for the sole purpose of getting clicks to generate ad revenue. They have no contact information, no social media, and no author bylines. Otherwise, generally such notifications are well received.

In the case of NCSC.nl I figured I would get a prompt reply and a quick fix. I got the prompt reply, but not the fix. First, note that they provide limited advisory services notifying their stakeholders of vulnerabilities and a page describing what the advisories are. They also have a PDF with a bigger explanation of what a security advisory is. Per Google translate, the advisories “… aim is to describe what the vulnerability is and what could possibly happen if it is exploited.” Simple and straight-forward. As most security professionals know, accuracy in an advisory is important. A typo in a CVE could point to the wrong vulnerability which might be the wrong software completely, or the right software and the wrong vulnerability. I contacted their info@ to let them know about the typo:

https://advisories.ncsc.nl/advisory?id=NCSC-2021-0840

[..] CVE-2021-3715 , CVE-2021-38160 , CVE-2021-4049 [Link]

That should be CVE-2021-40490 at the end.

Brian

The prompt reply I received the next morning was rather baffling. They ‘investigated’ the issue, confirmed I was correct, and wrote a 62 word reply over six lines instead of just fixing the single character that was missing.

Thank you for your e-mail. Hereby we confirm that we have received your
email and investigated the issue. We would like to thank you for your
friendly remark. However, we have decided not to update the
advisory as the CVE number is written correctly in other places in the
advisory.
Feel free to contact us again if there are any questions left.

I naturally questioned them on this odd reply and refusal to fix an inaccurate CVE identifier:

Yes, I have questions.

Why wouldn’t you correct a simple typo? More specifically, for a CVE ID that can cause confusion for security practitioners trying to ensure they have accurate vulnerability intelligence. Anyone reading your advisory may go down a proverbial rabbit hole trying to figure out what CVE-2021-4049 represents and waste considerable time.

Consider that that typo caused our team to respond trying to figure out what that ID represents. Fortunately, we have amazing vulnerability intelligence and it was fairly easy to deduce what happened.

Your apathy in this matter is staggering.

I hoped that an explanation, with a bit of shaming, might prompt them to just fix the single missing character. Nope…

Thank you for your e-mail. We appreciate your concerns. When the advisory
needs to be updated the typo will be corrected.

OK, but the advisory literally needs to be updated to fix the typo. This recursive excuse is just absurd. 21 word reply this time instead of a one character fix. They appreciate my concerns, but not enough to fix ONE CHARACTER.

It’s hard to have faith in Information Security when a national security center doesn’t understand the importance of accuracy and integrity. I hope organizations in the Netherlands are not relying on the NCSC.

The Rundown: CVE IDs & RESERVED Status

During the process of assigning a CVE ID, there is a time period between the assignment and the disclosure, and again between the disclosure and it becoming available on MITRE’s CVE site or NIST’s National Vulnerability Database (NVD). During this period, the ID will be shown as RESERVED.

First, it is important to note that when an ID is part of a CVE Numbering Authority (CNA) pool of IDs to potentially be assigned, it is shown in RESERVED status. If an ID is not assigned that year, it is then supposed to be moved to REJECT status the following year per CVE rules. Bit odd they say the reason for the rejection will “will most often be stated“; most often and not always? If a CNA other than MITRE assigns an ID and the researcher and/or vendor later publicly discloses the vulnerability, it may still show as RESERVED. This happens when the CNA fails to notify MITRE despite being stipulated in CNA rules. It can also happen if the CNA notifies MITRE but it slips through the cracks. Pretty simple right?

If MITRE assigns the ID to a researcher, it is a lot more likely to stay in RESERVED status after disclosure because the researcher who publicly discloses the vulnerability doesn’t always notify MITRE. You may ask why MITRE doesn’t open the CVE with details themselves if it is public, and that is a great question! The simple answer is, MITRE does not really monitor public sources for disclosures any longer. Back in the day they would monitor Bugtraq and NTBugtraq and encouraged researchers to just disclose directly to those mail lists. During that time, they also said they monitored four sources for new vulnerability information but notably did not include either mail list, instead including four different summaries being published. I think we can chalk that up to an error in documentation.

For those not familiar with MITRE’s coverage for CVE, consider that they no longer publish three lists of interest. As late as March 6, 2016, MITRE maintained lists of what they considered:

  • Full Coverage Sources” – “For nearly all issues disclosed by the source that could be associated with a CVE entry, there will be an associated CVE entry, regardless of the criticality of the issue. Although a source is named as Full Coverage, we purposely use the phrasing “nearly all issues disclosed” to allow the flexibility to potentially postpone coverage of minor issues.”
  • Partial Coverage Sources” – “The source will be actively monitored but issues will be processed and associated with CVE entries based on a variety of editorial judgments.”
  • Must-Have Products” – “All products listed are considered to be “must have.” This means that we will ensure that a CVE-ID is issued for any public disclosure for the product provided that the following to provisions are met…”

By the end of 2016, that page maintained the same URL but changed content to become what would be their CNA coverage page. By early 2017, the old URL redirected to a new one about requesting a CVE ID and CNA coverage, which is roughly the same as currently available. This is an important shift in how CVE operates as MITRE basically threw in the towel trying to actively monitor disclosures and moved to relying almost entirely on CNAs and researchers coming to them.

The part that is truly baffling to me is that this tax-payer funded project, costing us millions a year, thought that monitoring 48 sources for “full” coverage, 45 sources for “partial” coverages, and guaranteeing 45 products was ever adequate to begin with, and somehow a burden at that point. They also disclaimed that they “actively [monitor] many sources beyond this list. These sources include things like blogs from vulnerability researchers, conference proceedings, and media outlets.” Despite that claim and coverage, MITRE was already missing thousands of vulnerability disclosures a year including ones from sources on their list.

What should worry consumers of CVE is that other vulnerability databases monitor a lot more sources than that for a lot less money. Any claims of it being more complicated or the issue being due to their processes mean there is an incredible amount of red tape or horribly outdated technical processes that were never updated. If another database can monitor literally several thousand sources a week for a fraction of the price, it speaks to MITRE not evolving over the years. Whew, glad that wraps it up!

Sorry, one last thing. Like entries in REJECT status, we can’t trust entries in RESERVED status either. Based on above and how MITRE operates, we know there are bound to be quite a few vulnerabilities where a researcher requested an ID, published details, and did not notify MITRE. Their backwards choice of not monitoring sources for disclosures means a disclosure may sit in RESERVED status for some time. How long? I went poking around a bit for fun and found this one. At the time of this blog, CVE-2000-1253 is still in RESERVED status (archive).

The issue? That was disclosed in 2015, and likely earlier. The actual vulnerability details were public at far back as 2003, maybe earlier. The good news? If you aren’t worried about remote root on a medical device, no need to be worried about this one.

The Rundown: CVE IDs & REJECT Status

For analysts and practitioners that digest CVE regularly, you will likely be familiar with CVEs that are in REJECT status. If you are new to CVE or not familiar with some of the more gritty details, a CVE assignment may be rejected for various reasons. When that happens, it will receive a capitalized REJECT status:

The REJECT links to a page that offers more information, but as of April, 2021, actually links to the correct page but the wrong anchor. I’ll link to the correct anchor for your reference, which gives us several reason an ID might be rejected including “it being a duplicate CVE Record, it being withdrawn by the original requester, it being assigned incorrectly, or some other administrative reason.

At the time of this blog, there are almost 9,500 IDs that have been rejected. A significant portion of those come from MITRE being more proactive and enforcing that CVE Numbering Authorities (CNA) reject unused IDs from prior years, along with a general increase in the total CVE assigned per year.

The process of rejecting, and the presence of REJECT entries is straightforward.

That’s right, if I am taking the time to blog about a topic seemingly so easy, there’s probably more to it. In this case, I wanted to point out a couple examples of CVE IDs that are in REJECT status, but highlight issues. The first is a simple one that underscores that the process of CNAs rejecting CVE IDs may have a problem, or that MITRE has an issue in the way they described the rejected ID. We’ll take CVE-2018-1226 (archive), that was rejected because “The CNA or individual who requested this candidate did not associate it with any vulnerability during 2018. Notes: none.” That is easy enough, right? The problem is that it was rejected by March 19, 2018. Not even a quarter of the way through 2018 and it was rejected because it was not associated with a vulnerability that year? That seems problematic. I’m sure there is a good explanation for this, but the description sure doesn’t do it.

You may think that pointing that out is pedantic, and you are right. However, there is an important reason we need to be pedantic and expect accurate descriptions from CVE, even for a rejected entry. What if the REJECT message was factually incorrect? What if that CVE ID represented a valid vulnerability that impacted your organization? If you rely on CVE/NVD you would have a blinds pot as a result of errors in their process, which are critical to you. Looking at two older rejected CVE IDs as an example, CVE-2015-0788 (archive) and CVE-2015-0789 (archive), we see that both are in REJECT status because they were not associated with a vulnerability in 2015.

Looking closer, we can see that the assigning CNA was Micro Focus International. As such they should be the single source of truth and provenance for any vulnerability information associated with those CVEs. MITRE would be secondary and should not necessarily be trusted if there is a dispute. In this case, there is a dispute in the form of NetIQ Identity Manager release notes listing both CVEs as fixed issues in version 4.5 Service Pack 2.

NetIQ was founded in 1995, acquired by Attachmate in 2006, and then acquired by Micro Focus in 2014. With this document we see the conflict where Micro Focus says that they were assigned, represented legitimate vulnerabilities, and were fixed and CVE says they are rejected.

The takeaway here is that while a CVE may be listed as REJECTed, trust, but verify.

The Rundown: CVE IDs, Meanings, & Assumptions

For almost two decades, CVE has been considered an industry standard for vulnerability tracking. A CVE ID can be affiliated with many vulnerabilities, in a format like CVE-2014-54321. Note my choice in ID, from 2014 with a consecutive set of numbers. That is because I specifically chose a ‘sample’ CVE that was set aside as an example of the CVE ID Syntax Change in 2014. This change occurred when it was determined that 9,999 IDs for a single year was not going to be sufficient. Technical guidance on this is available, as well as more basic information and the announcement about the change. Starting out with this hopefully demonstrates that there may be more to an ID than meets the eye.

Fundamentally, the ID is simple; you have the CVE prefix, followed by a year identifier, and a numeric identifier. In the CVE used above, it would represent ID 54321 with a 2014 year identifier. Fairly simple! But you are reading an entire blog on these IDs by me so the spoiler is here. It isn’t so simple unfortunately. I want to give a rundown of what a CVE ID really is, and set the record straight. Why? Because I don’t think MITRE has done a good job with that, and worse, actively works against what could be a clear and simple policy. We’ll use CVE-YEAR-12345 as a representative example for the purpose of discussing these IDs to be clear about which part of an ID we’re talking about.


When CVE was started in 1999, assignments were made based on a public disclosure. However, from the beginning, the YEAR portion almost immediately was not guaranteed to represent the year of disclosure. This was because MITRE’s policy was to assign an ID for a pre-1999 vulnerability using a CVE-1999 ID. We can see this with CVE-1999-0145 which was assigned for the infamous Sendmail WIZ command, allowing remote root access. This feature was publicly disclosed as a vulnerability on November 26, 1983 as best I have determined (the Sendmail changelog). While it was a known vulnerability and used before that, it was privately shared. If there is a public reference to this vulnerability before that date, leave a comment please!

The takeaway is that a vulnerability from 1983 has a CVE-1999 identifier. So from the very first year, MITRE set a clear precedent that the YEAR portion of an ID does not represent the year of discovery or disclosure. You may think this only happened for vulnerabilities prior to 1999, but that isn’t the case. In the big picture, meaning the 22 years of CVE running, an ID typically does represent the disclosure year. However, per one of CVE’s founders, “because of CVE reservation, sometimes it aligned with year of discovery“. That is entirely logical and expected as a CVE ID could be used to track a vulnerability internally at a company before it was disclosed. For example, BigVendor could use the CVE ID not only for their internal teams, such as communicating between security and engineering, but when discussing a vulnerability with the researcher. If a researcher reported several vulnerabilities, using an ID to refer to one of them was much easier than the file/function/vector.

For the early CVE Numbering Authorities (CNA), companies that were authorized to assign a CVE without going through MITRE, this was a common side effect of assigning. If a researcher discovered a vulnerability on December 25 and immediately reported it to the vendor, it may be given e.g. a CVE-2020 ID. When the vendor fixed the vulnerability and the disclosure was coordinated, that might happen in 2021. The founder of CVE I spoke to told me there “weren’t any hard and fast rules for CNAs” even at the start. So one CNA might assign upon learning of the vulnerability while another might assign on public disclosure.

Not convinced for some reason? Let’s check the CVE FAQ about “year portion of a CVE ID”!

What is the significance and meaning of the YEAR portion of a CVE ID
CVE IDs have the format CVE-YYYY-NNNNN. The YYYY portion is the year that the CVE ID was assigned OR the year the vulnerability was made public (if before the CVE ID was assigned).

The year portion is not used to indicate when the vulnerability was discovered, but only when it was made public or assigned.

Examples:

A vulnerability is discovered in 2016, and a CVE ID is requested for that vulnerability in 2016. The CVE ID would be of the form “CVE-2016-NNNN”.
A vulnerability is discovered in 2015 and made public in 2016. If the CVE ID is requested in 2016, the CVE ID would be of the form “CVE-2016-NNNNN”.

All clear, no doubts, case closed!

That clear policy is conflicting or may introduce confusion in places. Looking at MITRE’s page on CVE Identifiers, we see that the “The process of creating a CVE Record begins with the discovery of a potential cybersecurity vulnerability.” My emphasis on ‘discovery’ as that means the ID would reflect when it was discovered, and not necessarily even when it was reported to the vendor. There are many cases where a researcher finds a vulnerability but may wait days, weeks, months, or even years before reporting it to the vendor for different reasons. So it is more applicable that the ID will be assigned based on when the vendor learns of the vulnerability in cases of coordinated disclosure with a CNA. Otherwise, a bulk of CVEs are assigned based on the disclosure year.

It gets messier. At the beginning of each year, each CNA will get a pool of CVE IDs assigned. The size of the pool varies by CNA and is roughly based on the prior year of assignments. A CNA that disclosed 10 vulnerabilities in the prior year is likely to get 10 – 15 IDs the subsequent year. Per section 5.1.4 of the CNA rules, any IDs that are not used in a calendar year should be REJECTed if they were not assigned to an issue. “Those CVE IDs that were unused would be rejected.” But then, it stipulates that the CNA can get “CVE IDs for previous calendar years can always be requested if
necessary.
” So per current rules, a CNA can request a new ID from a prior year despite REJECTing IDs that were previously included in their pool. That means it is entirely optional, up to each CNA, on how they assign.

[Update: Note that the pool of IDs a CNA gets one year may not be the same the next. Not only in regards to the size of the pool, but the first ID may be in an entirely different range. e.g. 2019-1000 vs 2020-8000.]

The take-away from all this is that we now have many reasons why a CVE ID YEAR component does not necessarily tie to when it was disclosed. The more important take-away? If you are generating statistics based on the YEAR component, you are doing it wrong. Any statistics you generate are immediately inaccurate and cannot be trusted. So please don’t do it!


Finally, a brief overview of the numeric string used after the YEAR. Going back to our example, CVE-YEAR-12345, it is easy to start to make assumptions about 12345. The most prevalent assumption, and completely incorrect, is that IDs are issued in a sequential order. This is not true! Covered above, CNAs are given pools of IDs at the beginning of each year. Oracle and IBM assign over 700 vulnerabilities a year, so the pool of IDs they receive is substantial. There are over 160 participating CNAs currently, and if each only received 100 IDs, that is over 16,000 IDs that are assigned before January 1st.

In 2021, the effect of this can be seen very clearly. Halfway through April and we’re already seeing public IDs in the 30k range. For example, CVE-2021-30030 is open and represents a vulnerability first disclosed on March 28th. According to VulnDB, there are only 7,074 total vulnerabilities disclosed this year so far. That means we can clearly see that CVE IDs are not assigned in order.

Redscan’s Curious Comments About Vulnerabilities

As a connoisseur of vulnerability disclosures and avid vulnerability collector, I am always interested in analysis of the disclosure landscape. That typically comes in the form of reports that analyze a data set (e.g. CVE/NVD) and draw conclusions. This seems straight-forward but it isn’t. I have written about the varied problems with such analysis many times in the past and yet, companies that don’t operate in the world of vulnerability databases still decide to play in our mud puddle. This time is the company Redscan, who I don’t think I had heard of, doing analysis on NVD data for 2020. Risk Based Security wrote a commentary on their analysis, to which I contributed, but I wanted to keep the party going over here with a few more personal comments. Just my opinions here, as a more outspoken critic on the topic, and where I break from the day job.

I am going to focus on one of my favorite topics; vulnerability tourists. People that may be in the realm of Information Security, but don’t specifically operate day-to-day in the world of vulnerability disclosures, and more specifically to me, vulnerability databases (VDBs). For this blog, I am just going to focus on a few select quotes that made me double-take. Read on after waving to Tourist Lazlo!

“The NVD tracks CVEs logged by NIST since 1988, although different iterations of the NVD account for some variation when comparing like-for-like results over time.”

There’s a lot to unpack here, most of it wrong. First, the NVD doesn’t track anything; they are spoon-fed that data from MITRE, who manages the CVE project. Second, NIST didn’t even create NVD until over five years after CVE started. Third, CVE didn’t track vulnerabilities “since 1988”; they cherry-picked some disclosures from before 1999, when they started, and why CVE IDs start with ‘1999’. Fourth, there was only one different iteration of NVD, that was their ICAT “CVE Metabase” that ran the first year of CVE basically. According to Peter Mell, who created it, said that after starting as its own vulnerability website, “ICAT had become an archival tool for CVE standard vulnerabilities and was only updated every three or four weeks”. Then in 2005 the site relaunched with a new focus and timely updates from CVE. Despite this quote, later in their report they produce a chart that tries to show an even comparison from 1988 to 2020 despite saying it went through iterations and despite not understanding CVSS.

“The growth is also likely attributable to an increase in the number of CVE Numbering Authorities (CNAs) – of which there are now more than 150 worldwide with the power to create and publish CVEs.”

The growth in disclosures aggregated by CVE is a lot more complicated than that, and the increase in CNAs I doubt is a big factor. Of course, they say this and don’t cite any evidence despite CVE now showing who the assigning CNA was (e.g. CVE-2020-2000 is Palo Alto Networks). The data is there if you want to make that analysis but it isn’t that easy since it isn’t included in the NVD exports. That means it requires some real work scraping the CVE website since they don’t include it in their exports either. Making claims without backing them up when the data is public and might prove your argument is not good.

“Again, this is a number that will concern security teams, since zero interaction vulnerabilities are famously difficult to detect and have the potential to cause significant damage.”

This makes me think that Redscan should invent a wall, perhaps made of fire, that could detect and prevent these attacks! Or maybe a system that is designed to detect intrusions! Or even one that can prevent intrusions! This quote is one that is truly baffling because it doesn’t really come with an explanation as to what they mean, and I hope they mean something far different than what this sounds like. I hope this isn’t a fear tactic to make readers think that their managed detection service is needed. Quite the opposite; anyone who says the above probably should not be trusted to do your attack detection.

This chart heading is one of many signs that Redscan doesn’t understand CVSS at all. For a “worst of the worst” vulnerability they got several attributes right but end up with “Confidentiality [High]”. The vulnerability they describe would only be CVSSv2 7.8 (AV:N/AC:L/Au:N/C:C/I:N/A:N) and CVSSv3.1 7.5 (AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). That is not the worst. If it ‘highly’ impacts confidentiality, integrity, and availability then that becomes the worst of the worst, becoming CVSSv2 10.0 and CVSSv3.1 9.8 or 10.0 depending on scope. It’s hard to understand how a security company gets this wrong until you read a bit further where they say they “selected the very ‘worst’ option for every available metric.” My gut tells me they didn’t realize you could toggle ‘High’ for more than one impact and confidentiality is the first listed.

“It is also important to note that these numbers may have been artificially reduced. Tech giants such as Google and Microsoft have to do a lot to maintain their products and services day-to-day. It is common for them to discover vulnerabilities that are not being exploited in the wild and release a quick patch instead of assigning a CVE. This may account for fewer CVEs with a network attack vector in recent years.”

This is where general vulnerability tourism comes in as there is a lot wrong here. Even if you don’t run a VDB you should be passingly familiar with Microsoft advisories, as an example. Ever notice how they don’t have an advisory with a low severity rating? That’s because they don’t publish them. Their advisories only cover vulnerabilities at a certain threshold of risk. So that means that the statement above is partially right, but for the wrong reason. It isn’t about assigning a CVE, it is about not even publishing the vulnerability in the first place. Because they only release advisories for more serious issues, it actually skews their numbers to include more remote vulnerabilities, not less, primarily on the back of “remote” issues that require user interaction such as browser issues or file parsing vulnerabilities in Office.

This quote also suggests that exploitation in the wild is a bar for assigning a CVE, when it absolutely is not. It might also be a surprise to a company like Redscan, but there are vulnerabilities that are disclosed that never receive a CVE ID.

“Smart devices designed for the mass market often contain a worrying number of vulnerabilities due to manufacturer oversight. Firmware within devices is often used by multiple vendors, meaning that any vulnerabilities in this software has the potential to result in lots of CVEs.”

Wrong again, sometimes. If it is known to be the same firmware used in multiple devices, it gets one CVE ID. The only time there are additional IDs assigned is when multiple disclosures don’t positively ID the root cause. When three disclosures attribute the same vulnerability to three different products, it stands to reason there will be three IDs. But it isn’t how CVE is designed because it artificially inflates numbers, and that is the game of others.

“The prevalence of low complexity vulnerabilities in recent years means that sophisticated adversaries do not need to ‘burn’ their high complexity zero days on their targets and have the luxury of saving them for future attacks instead.”
-vs-
“It is also encouraging that the proportion of vulnerabilities requiring high-level privileges has been on the increase since 2016. This trend means that cybercriminals need to work harder to conduct their attacks.”

So which is it? When providing buzz-quote conclusions such as these, that are designed to support the data analysis, they shouldn’t contradict each other. This goes back to what I have been saying for a long time; vulnerability statistics need qualifications, caveats, and explanations.

“Just because a vulnerability is listed in the NVD as hard to exploit doesn’t mean that attackers aren’t developing PoC code to exploit it. The key is to keep up with what’s happening in the threat landscape and respond accordingly.”

I’ll end here since this is a glowing endorsement for why vulnerability intelligence has to be more evolved than what CVE and NVD are offering. Part of the CVSS specifications include Temporal scoring and one of those attributes is Exploit Code Maturity. This is designed to specifically address the problem above; that knowing the capability of potential attackers matters. With over 21,000 vulnerabilities disclosed last year, organizations are finding that just patching based on the CVSSv3 base score isn’t enough. Sure, you patch the 10.0 / 9.8 since those are truly the worst-of-the-worst, and you patch the 9.3 / 8.8 since any random email might carry a payload. Then what? If all things are equal between vulnerabilities that impact your organization you should look to see if a patch is available (also covered by Temporal score) and if an exploit is available.

Numeric scores are not enough, you have to understand the context behind them. That CVSSv2 remote information disclosure that partially affects confidentiality by disclosing an admin password is only a 5.0. Score it under CVSSv3 and you are looking at a 9.8 because it immediately leads to privilege escalation which is factored in under that system. Heartbleed was a CVSSv2 5.0 with a functional exploit and available patch; look what hell that brought upon us. If you aren’t getting that type of metadata, reconsider your choice of vulnerability intelligence.

A critique of the summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”

Update: Corren McCoy has written a wonderful response to this blog where she goes into more detail about her conclusions as well as citing more portions of the original research that led to her conclusions. As she notes, there are several layers of condensing the original research at play here, which can dilute and distort the original research. In her follow-up she breaks down each of these areas that I address below. If you continue reading my blog below, please read her reply after to get a better picture. Thank you!


What do you think of this?” It always starts out simple. A friend asked this question of an article titled Summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”. This study is math heavy and that is not my jam. But vulnerability databases are, and that includes the CVE ecosystem which encompasses NVD. I am also pretty familiar with limitations of the CVSS scoring system and colleagues at RBS have written extensively on them.

I really don’t have the time or desire to dig into this too heavily, but my response to the friend was “immediately problematic“. I’ll cliff notes some of the things that stand out to me, starting with the first graphic included which she specifically asked me about.

  • The header graphic displays the metrics for the CVSSv3 scoring system, but is just labeled “CVSS”. Not only is this sloppy, it belies an important point of this summary that the paper’s work is based on CVSSv2 scores, not CVSSv3. They even qualify that just below: “We should note the analysis conducted by Ross et al. is based upon the CVSS Version 2 scoring system…
  • Ross et al. note that many exploits exist without associated CVE-IDs. For example, only 9% of the Symantec data is associated with a CVE-ID. The authors offered additional caveats related to their probability calculation.” That sounds odd, but it is readily explained above when they summarize what that data is: “Symantec’s Threat Database (SYM): A database extracted from Symantec by Allodi and Massacci that contains references to over 1000 vulnerabilities.” First, that data set contains a lot more than vulnerabilities. Second, if Symantec is really sitting on over 900 vulnerabilities that don’t have a CVE ID, then as a CNA they should either assign them an ID or work with MITRE to get an ID assigned. Isn’t that the purpose of CVE?
  • Ross et al. use four datasets reporting data on vulnerabilities and CVSS scores…” and then we see one dataset is “Exploit Database (Exploit-DB): A robust database containing a large collection of vulnerabilities and their corresponding public exploit(s).” Sorry, EDB doesn’t assign CVSS scores so the only ones that would be present are ones given by the people disclosing the vulnerabilities via EDB, some of whom are notoriously unreliable. While EDB is valuable in the disclosure landscape, serving as a dataset of CVSS scores is not one of them.
  • About 2.7% of the CVE entries in the dataset have an associated exploit, regardless of the CVSS V2 score.” This single sentence is either very poorly written, or it is all the evidence you need that the authors of the paper simply don’t understand vulnerabilities and disclosures. With a simple search of VulnDB, I can tell you at least 55,280 vulnerabilities have a CVE and a public exploit. There were 147,490 live CVE IDs as of last night meaning that is almost 38% that have a public exploit. Not sure how they arrived at 2.7% but that number should have been immediately suspect.
  • In other words, less than half of the available CVSS V2 vector space had been explored despite thousands of entries…” Well sure, this statement doesn’t qualify one major reason for that. Enumerate all the possible CVSSv2 metric combinations and derive their scores, then look at which numbers don’t show up on that list. A score of 0.1 through 0.7 is not possible for example. Then weed out the combinations that are extremely unlikely to appear in the wild, which is most that have “Au:M” as an example, and it weeds out a lot of possible values.
  • Only 17 unique CVSS vectors described 80% of the NVD.” Congrats on figuring out a serious flaw in CVSSv2! Based on the 2.7% figure above, I would immediately question the 80% here too. That said, there is a serious weighting of scores primarily in web application vulnerabilities where e.g. an XSS, SQLi, RFI, LFI, and limited code execution could all overlap heavily.
  • Input: Vulnerabilities (e.g., NVD), exploit existence, (e.g., Exploit-DB), the number of clusters k” This is yet another point where they are introducing a dataset they don’t understand and make serious assumptions about. Just because something is posted to EDB does not mean it is a public exploit. Another quick search of VulnDB tells us there are at least 733 EDB entries that are actually not a vulnerability. This goes back to the reliability of the people submitting content to the site.
  • The authors note their approach outperforms CVSS scoring when compared to Exploit-DB.” What does this even mean? Exploit-DB does not do CVSS scoring! How can you compare their approach to a site that doesn’t do it in the first place?

Perhaps this summary is not well written and the paper actually has more merit? I doubt it, the summary seems like it is comprehensive and captures key points, but I don’t think the summary author works with this content either. Stats and math yes. Vulnerabilities no.

Not all CVEs are Created Equal. Or even valid…

[I wrote this early 2019 and it was scheduled for January 7 but it apparently did not actually publish and then got lost in my excessive drafts list. I touched it up this week to publish because the example that triggered this blog is old but the response is evergreen. Apologies for the long delay!]

I recently caught a Tweet from @NullCon offering 10 free conferences passes to NullconDasham, awarded to “InfoSec heroes who shared their hard work with the community & contributed to the @CVEnew database“.

I re-Tweeted with comment pointing out “There were at least 311 CVE assignments in 2018 alone, that were for issues that were *not* a vulnerability. I hope you are going to scrutinize the submissions.” Anant Shrivastava replied asking for some examples, and the next morning Mitja Kolsek suggested a blog would be beneficial. Here it is, as short and sweet as I can, which is never short in the world of Vulnerability Databases (VDBs) due to caveats.

Continue reading

More authorities, more CVEs; Oh, and more commentary.

On November 10, TechBeacon published a great article by Rob Lemos titled “More authorities, more CVEs: What it means for app sec teams” in which I was quoted, along with several other people.

Like many articles of this nature, those who provide input often will talk for as long as half an hour and ultimately get a couple lines quoted. We do it to provide background and context on the topic as well as have an open discussion on vulnerability trends. That means there are ‘outtake’ opinions and facts, as well as our potential reaction to other parts of the article that did not include our input. So this blog just covers some of my random observations to compliment the article.

Until 2016, more than 80% of software security issues assigned a CVE identifier belonged to only 10 classes, or weaknesses, as classified by their Common Weakness Enumeration (CWE) category. But in 2019, the top 10 weaknesses only accounted for 59% of reported vulnerabilities.

The Common Weakness Enumeration (CWE) index is interesting to me and I wonder if it has gotten so big to degrade its value. Consider that there are now 891 CWE identifiers as of August 20 in version 4.2 of the framework. Per the article, only 10 of them account for 59% of vulnerabilities which will no doubt include XSS, SQLi, and CSRF as examples. That makes me wonder the value of abstracting so much as it means that hundreds of those CWEs will represent a handful of vulnerabilities at most.

Digging into the 2,298 page PDF documenting version 4.2, you can jump toward the end of the CWE list and see that several have been created but have no “Observed Examples”. In fact, searching for that phrase only yields 397 hits. Does that mean that out of 891 CWE IDs representing weaknesses, that MITRE has only come up with 397 that match known vulnerabilities? I certainly expect otherwise and hope this is just documentation shortcoming as I feel that every CWE ID should be linked to a concrete real-world example. I’d love to see

I’d love to see a simple breakdown of the top 100 CWE along with how many vulnerabilities are associated with them (via NVD, since MITRE doesn’t actually apply CWE to entries) and what percentage of the overall vulnerabilities that represents. It might be very telling just how useful CWE is and if the project is being pushed too heavily from an academic standpoint. Before you judge that comment, let me know how useful this CWE report from MITRE is, and make sure you load it in Chrome.


It’s an open question whether the addition of coding repositories will lead to another expansion in the number of vulnerabilities.

I don’t think that is an open question at all. I think the number of vulnerabilities will go up as a result of more coding repositories becoming a CNA. But that isn’t really the issue here. Instead, the real questions should be centered around what quality of CVE entries they will provide, if they will adhere to CNA standards, and if MITRE will enforce CNA policy on them.

Based on history, the short answers to those questions are: quality will go down, no, and no. As soon as MITRE provides a breakdown of how many IDs were published by each CNA it is difficult to know. Speaking of, why hasn’t MITRE published such statistics? Rhetorical question, apologies.


Open source vulnerabilities: Do you know what’s in your software?

I, along with many others, can’t stress this enough! Please make sure you understand what third-party software your developers are using. This affects your organizations from both a vulnerability standpoint, as well as legal accountability. Using a third-party library against its license could open you up to some hardships.

The only reason I quoted this section is because I just read an article in the latest Wired that mentions Bootstrap is thought to be used on nearly 20% of all web sites across the Internet. That is incredible.


Patch Tuesday is becoming a bottleneck

There is a lot more that can be said on this topic. It reminds me of a 2015 blog I wrote that actually goes back farther to 2007 where this problem was predicted long before the nightmare IT teams experience once a month. It’s only going to get worse as more vendors jump on this patch schedule and the only ones who will suffer are their paying customers. But hey, let’s let them keep using the term “responsible” disclosure too.


But exploitability alone doesn’t solve the problem—three quarters of the 17,300 vulnerabilities identified ranked CVSS exploitability rating of 8.0 or higher.

I’m not sure a more perfect example of why CVSS has become worthless exists. On its own, especially using the Base score only, is it really helpful that so many vulnerabilities are ‘High Risk‘? This is also a good reminder of another blog I have been meaning to write for a while that outlines the distribution of CVSSv2 versus CVSSv3 and how it impacts scoring. With a couple charts you will get a nice visual of just how poorly thought out some of the framework was. Of course, this has already been evaluated by others years back as well.


Finally, because I don’t hold the copyright to the picture used in the TechBeacon article header, I offer my version:

Picture of me by D2d in November, 2018 at Tomayo, in Denver, CO.

“The History of CVE” and A Couple of Objections

I just read “The History of Common Vulnerabilities and Exposures (CVE)” by Ary Widdes from Tripwire and found it to be a great summary of the 20+ years of the program. I say that as an outspoken CVE and MITRE critic even! I do have a couple of objections however, with the conclusion, and then a fun bounty!

Widdes concludes the history by saying:

A lot has changed in the 21 years since the CVE List’s inception – both in terms of technology and vulnerabilities. Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage. It’s also possible that someone else would have created a service similar to the CVE List. Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

There’s a lot to unpack here so I will take it one sentence at a time, starting with the second.

“Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage.”

No, there is no “possible” here. That is a simple reality with an important caveat. The reality is that teams of all types still use multiple tools from multiple vendors to do their job. The caveat, and more to the point of that sentence, is that CVE doesn’t offer “complete coverage” and many of the vulnerability scanners only cover a third of the issues in CVE for various reasons. Even using a combination of firewalls, vulnerability scanners, intrusion detection/prevention, audits, and a slew of other tools, organizations are likely seeing half of what CVE has to offer at best. Widdes’ conclusion here gives undue credit to CVE and the state of vulnerability coverage it offers.

It’s also possible that someone else would have created a service similar to the CVE List.

This is where the vulnerability historian in me wants to rage a bit. This statement is unequivocally false for the simple reason that vulnerability databases existed before CVE, both free (e.g. X-Force) and commercial (e.g. RSI), in 1997 alone [1]. The first vulnerability database was created in 1973, specific to Multics, but also when there weren’t that many other systems to catalog bugs or vulnerabilities in. In 1983 we saw the Mt Xinu Bug List and in 1985 Matt Bishop’s List of UNIX Holes, both of which were more comprehensive than one platform. If we consider a vulnerability database implemented via product, we had ISS, SATAN, Ballista, and Nessus between 1995 and the creation of CVE in 1999. Many of the hackers turned security professionals may fondly remember Fyodor’s Exploit World (1996 – 1998) from both aspects of their lives. Those same folks probably also remember Packet Storm (1998) which is still running today.

Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

This, unfortunately, is true. I say unfortunately because of my long-standing criticisms of CVE over the past decade, but won’t go into here.

Bug(s) Bounty:

If there is anyone at MITRE open to outright bribery, including all-you-can-eat sushi dinners, I will pay a bounty to get my hands on that list of 8,400 submissions! While I know there are likely a lot of duplicates, the vulnerability historian in me would love to audit that data to see if MITRE decided to skip any that would be considered vulnerabilities by today’s standards, or where someone else back then had more knowledge of a vulnerability than was submitted. That data is over twenty years old and was solicited, processed, and partially published with U.S. taxpayer funded money. There’s no reason not to make it public. =)

[1] The Repent Security Inc. (RSI) database existed in 1997 but may not have been offered as a commercial product until 1998.

Why @anacondainc Doesn’t Fully Understand CVEs

It’s worrisome that in 2020 we still have people in influential technical roles that don’t understand CVE. A friend told me earlier this year he was in a meeting where someone said that CVE IDs are assigned in order, so CVE-2020-9500 meant there were 9500 vulns in 2020 so far. Of course that is not how it works and a dangerous understanding of CVE.

I ran across an article written by Nick Malkiewicz of Anaconda titled “Why Understanding CVEs Is Critical for Data Scientists“. This article has several bits that show a lack of understanding of what CVE is. One of the biggest is equivocating a CVE with a vulnerability. Yes, many vulnerabilities directly map to a single CVE identifier, but a CVE is the identifier not the vulnerability. Additionally, sometimes one vulnerability can track with multiple CVE IDs, or one CVE ID can track to multiple vulnerabilities. So lines like the following are concerning:

When someone finds a CVE, they report it to a CVE Numbering Authority (CNA).

When someone finds a vulnerability, they report it to MITRE or a vendor, who may be a CNA but more often not one. That vendor can then ask MITRE for an ID via a web form.

CNAs assign identification numbers to CVEs and list them in publicly accessible databases.

A CNA is required to inform MITRE after a CVE-assigned vulnerability has been disclosed. That is actually a fairly recent rule, implemented in the last few years. For most of CVE’s history there was no requirement or specific communication channel for a CNA to notify MITRE of this. That was one of many failings of the CVE ecosystem and directly led to companies being breached, as they relied on CVE to be ‘complete’ and timely.

Each vulnerability listed in a CVE database has a score from .1 to 10, 10 being the highest risk level. These scores are based on exploitability, impact, remediation level, report confidence, and other qualities.

Technically, not even the first line is true as NVD can score a vulnerability as 0.0, meaning it is not a vulnerability and poses no risk. This occurs when a researcher or vendor disclose a vulnerability but don’t fully understand the issue or the subsequent impact. This happens hundreds of times a year although many are not included in NVD. The second sentence from Anaconda is also incorrect as NVD only scores CVSS Base scores. The exploitability, remediation level, and report confidence are part of Temporal scores and not included. You can see an example with CVE-2020-2800 published by Oracle and given a CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N score by both Oracle and NVD. This misunderstanding of NVD CVSS scoring is more baffling as Anaconda links to the same FIRST CVSS document I do in this paragraph.

Anaconda goes on talking about how there are other factors at play including looking at the history of a package, how fast vendors respond, and more. This is great advice and critical for companies facing tens of thousands of vulnerabilities a year. Unfortunately, they slide into the “more lipstick on that pig” problem:

The good news is, there are tools that automate the CVE monitoring process.

This is true. But, more ways to manipulate bad data still leaves you with bad data. In addition to CVE missing several thousand vulnerabilities a year, their push for quantity in the last few years have led to a serious hit on quality. There are some CVE IDs that have descriptions missing critical information like the vendor, affected version, or impact. All the data wrangling and fancy analysis of that data is still based on bad or incomplete information. All the lipstick on that pig still makes it a pig.

Finally, I will quote on other line from their blog that is curious:

Hacking open-source software also has a bigger payoff because many more people use it.

I understand and appreciate the sentiment, and finding a vulnerability in a library like OpenSSL obviously has a huge impact. However, that and a couple dozen libraries are still the outliers in the bigger picture. Compare your vulnerabilities like EternalBlue to a standard open source library vulnerability and they are not even close as far as how “many more people use it”.