A critique of the summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”

What do you think of this?” It always starts out simple. A friend asked this question of an article titled Summary of “Latent Feature Vulnerability Rankings of CVSS Vectors”. This study is math heavy and that is not my jam. But vulnerability databases are, and that includes the CVE ecosystem which encompasses NVD. I am also pretty familiar with limitations of the CVSS scoring system and colleagues at RBS have written extensively on them.

I really don’t have the time or desire to dig into this too heavily, but my response to the friend was “immediately problematic“. I’ll cliff notes some of the things that stand out to me, starting with the first graphic included which she specifically asked me about.

  • The header graphic displays the metrics for the CVSSv3 scoring system, but is just labeled “CVSS”. Not only is this sloppy, it belies an important point of this summary that the paper’s work is based on CVSSv2 scores, not CVSSv3. They even qualify that just below: “We should note the analysis conducted by Ross et al. is based upon the CVSS Version 2 scoring system…
  • Ross et al. note that many exploits exist without associated CVE-IDs. For example, only 9% of the Symantec data is associated with a CVE-ID. The authors offered additional caveats related to their probability calculation.” That sounds odd, but it is readily explained above when they summarize what that data is: “Symantec’s Threat Database (SYM): A database extracted from Symantec by Allodi and Massacci that contains references to over 1000 vulnerabilities.” First, that data set contains a lot more than vulnerabilities. Second, if Symantec is really sitting on over 900 vulnerabilities that don’t have a CVE ID, then as a CNA they should either assign them an ID or work with MITRE to get an ID assigned. Isn’t that the purpose of CVE?
  • Ross et al. use four datasets reporting data on vulnerabilities and CVSS scores…” and then we see one dataset is “Exploit Database (Exploit-DB): A robust database containing a large collection of vulnerabilities and their corresponding public exploit(s).” Sorry, EDB doesn’t assign CVSS scores so the only ones that would be present are ones given by the people disclosing the vulnerabilities via EDB, some of whom are notoriously unreliable. While EDB is valuable in the disclosure landscape, serving as a dataset of CVSS scores is not one of them.
  • About 2.7% of the CVE entries in the dataset have an associated exploit, regardless of the CVSS V2 score.” This single sentence is either very poorly written, or it is all the evidence you need that the authors of the paper simply don’t understand vulnerabilities and disclosures. With a simple search of VulnDB, I can tell you at least 55,280 vulnerabilities have a CVE and a public exploit. There were 147,490 live CVE IDs as of last night meaning that is almost 38% that have a public exploit. Not sure how they arrived at 2.7% but that number should have been immediately suspect.
  • In other words, less than half of the available CVSS V2 vector space had been explored despite thousands of entries…” Well sure, this statement doesn’t qualify one major reason for that. Enumerate all the possible CVSSv2 metric combinations and derive their scores, then look at which numbers don’t show up on that list. A score of 0.1 through 0.7 is not possible for example. Then weed out the combinations that are extremely unlikely to appear in the wild, which is most that have “Au:M” as an example, and it weeds out a lot of possible values.
  • Only 17 unique CVSS vectors described 80% of the NVD.” Congrats on figuring out a serious flaw in CVSSv2! Based on the 2.7% figure above, I would immediately question the 80% here too. That said, there is a serious weighting of scores primarily in web application vulnerabilities where e.g. an XSS, SQLi, RFI, LFI, and limited code execution could all overlap heavily.
  • Input: Vulnerabilities (e.g., NVD), exploit existence, (e.g., Exploit-DB), the number of clusters k” This is yet another point where they are introducing a dataset they don’t understand and make serious assumptions about. Just because something is posted to EDB does not mean it is a public exploit. Another quick search of VulnDB tells us there are at least 733 EDB entries that are actually not a vulnerability. This goes back to the reliability of the people submitting content to the site.
  • The authors note their approach outperforms CVSS scoring when compared to Exploit-DB.” What does this even mean? Exploit-DB does not do CVSS scoring! How can you compare their approach to a site that doesn’t do it in the first place?

Perhaps this summary is not well written and the paper actually has more merit? I doubt it, the summary seems like it is comprehensive and captures key points, but I don’t think the summary author works with this content either. Stats and math yes. Vulnerabilities no.

Commentary on Radware’s Top Web Exploits of 2020

At the close of each year we see at least one article covering the top vulnerabilities / exploits from the prior year. This is usually written on the back of having large detection networks across the Internet that get a comprehensive view of exploitation. It’s a great way to get real intelligence for criminal hacking activity. Unfortunately, we often see a breakdown when it comes to conveying that information in a useful manner. I know there is an argument to be made that the companies releasing such blogs are primarily after PR, sure. But they also have an opportunity to help their clients and the rest of the world by ensuring the blogs contain more useful and actionable information.

For this commentary, I’ll examine Radware’s blog, “The Top Web Service Exploits in 2020” published December 23, 2020 and covered almost verbatim by Security Magazine on January 5, 2021. I don’t have a view into exploit activity itself, but I do have a good view into the vulnerability disclosure landscape that is a cornerstone of this commentary.

We’ll start by setting a few basic ideas for mutual understanding for any such blog. First, each exploit should be tied to a unique vulnerability or it should explain it is an exploit chain and clearly delineate each vulnerability in the chain or explain what it represents if not a pure vulnerability. Second, it should provide at least one external reference for each vulnerability; either a CVE ID, vendor advisory, or commonly accepted third-party advisory such as US-CERT or another similar body. This is what allows the reader to quickly determine if their organization has patched against the vulnerability or not. If I have to spend considerable time trying to determine which vulnerability is being described, many organizations may be at a complete loss trying to figure it out.

With that, let’s look at the top 10 exploited vulnerabilities in 2020, according to Radware, and try to figure out some additional information for perspective. I will also be very clear that Radware’s blog is extremely frustrating and not immediately helpful, instead requiring a lot of extra work. The fact that they only attributed three exploits to a CVE ID is a dismal commentary on the CVE ecosystem. This analysis of their analysis will server as a reminder that comprehensive vulnerability intelligence is the foundation of any good security program.


Service Exploit #1: /ws/v1/cluster/apps/new-application

Based on their description, this appears to match VulnDB 184750 “Apache Hadoop YARN ResourceManager REST API Request Handling Remote Command Execution“. The first thing of interest is it was disclosed on October 19, 2016 and does not have a CVE assignment over four years later. No wonder many organizations aren’t aware of this vulnerability and have not sought out their own remediation strategy.

Service Exploit #2: /manager/html

This is summarized as “Apache Tomcat Manager Application Upload Authenticated Code Execution” and goes on to describe it as “This module can be used to execute a payload on Apache Tomcat servers that have an exposed “manager” application. The payload is uploaded as a WAR archive containing a JSP application using a POST request against the /manager/html/upload component.

Despite this description, that does not cleanly map to any vulnerability in VulnDB. The closest matches are CVE-2017-12615 and CVE-2017-12617 which is an abstraction for different platforms, but fundamentally “Apache Tomcat HTTP PUT Method JSP File Upload Remote Code Execution“. On the surface this is a match with Apache Tomcat, JSP application, and POST request to achieve code execution. However, those two CVEs cover a JSP file upload, not a WAR archive, and do not mention the /manager/html/upload component. So we’re left wondering if the exploit described is simply a misconfiguration scenario (i.e. intended functionality not secured) or an actual disclosed vulnerability.

Service Exploit #3: /level/15/exec/-/sh/run/CR

Based on the description, this is a misconfiguration scenario where an administrator sets up a Cisco router with the HTTP admin interface enabled, but without password protection. This allows an attacker to use the legitimate functionality to run arbitrary commands.

Service Exploit #4: /admin/assets/js/views/login.js

Radware says this “resource belongs to Sangoma FreePBX code and it looks like the attackers are trying to detect vulnerable FreePBX servers and exploit one of the known vulnerabilities.” The first issue is that script doesn’t immediately track to a VulnDB entry based on titles, which reflect the script name typically. However, let’s consider the URL being seen: … login.js. Rather than attempting to exploit “one of the known vulnerabilities“, I would suggest instead they are trying default credentials. At least back around 2000, the tried-and-true default credentials of admin/admin were all you needed to access the interface.

This one is curious to me because presumably a company that was detecting exploit traffic and could see e.g. POST requests as demonstrated in Service Exploit #2, would also see that the attackers were trying the default credentials. So we’re left with Service Exploit #4 being of little help and only creating confusion over what is being exploited.

Service Exploit #5: /ftptest.cgi?loginuse=&loginpas=

Radware attributes this to “many cheap Wireless IP web cameras use the same genetic code based on the GoAhead code (the tiny, embedded web server).” This tracks cleanly with VulnDB 181032 “Axis Multiple Products axis-cgi/ftptest.cgi Multiple Parameters Remote Command Execution Weakness“. This is actually a fun rabbit hole as this disclosure originally comes from an audit of a AXIS A1001 Network Door Controller and exploitation of this issue requires privileged access to the management interface. With that in mind, we’re back to a default credential scenario that may be the actual issue. Back in 2001, defaults for Axis network cameras were covered by CVE-2001-1543.

[Update: Z Balazs points out that this finding is likely due to Persirai botnet activity and links to more information.]

Service Exploit #6: /service/extdirect

This is the only one of the ten exploits covered that they include a CVE ID for. CVE-2019-7238 maps to VulnDB 198437 “Nexus Repository Manager /service/extdirect Insufficient Access Control Request Handling Remote Code Execution“. But, is that really the right ID? If we look at CVE-2020-10204 we are given a very brief summary of “Sonatype Nexus Repository before 3.21.2 allows Remote Code Execution” and a link to the vendor advisory. However, VulnDB 226228 also maps to this and is summarized as “Nexus Repository Manager /service/extdirect Request Handling Remote Command Execution“. We immediately see the /service/extdirect from Radware’s finding in both titles. The vendor’s advisory does not include this endpoint though, but we find it in this exploit published on GitHub that tracks with the CVE-2020-10204 and we see it in a different exploit for CVE-2019-7238.

CVE-2019-7238 was fixed in Nexus Repository Manager version 3.15.0 and CVE-2020-10204 was fixed in version 3.21.2. Due to the vague vendor advisories it difficult to tell if this was a regression situation or something else. But, the CVE-2020-10204 vendor advisory gives us the interesting bit in the context of exploitation: “The vulnerability allows for an attacker with an administrative account on NXRM to execute arbitrary code by crafting a malicious request to NXRM.” That is an important distinction! So this is likely CVE-2019-7238 as Radware says, unless there are default credentials which would allow for exploiting CVE-2020-10204 as well.

Looking at the NVD entry for CVE-2020-10204 we also see that they scored this incorrectly for their CVSSv3 score, as ‘Privileges Required‘ should be ‘High‘, notLow‘ as they have it.

Service Exploit #7: /solr/admin/info/system?wt=json

For this one, we get an Apache Bug ID (SOLR-4882) and CVE-2013-6397 as references which is great. That said, it would be very helpful if Radware would link to these resources to make it easier for their readers.

Service Exploit #8: /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php

This is the third exploit they match to an ID, CVE-2017-9841 and it was disclosed June 27, 2017. Another good reminder that software with disclosed vulnerabilities and vendor solutions are not being applied, causing many organizations to become low-hanging fruit in the exploit world.

One little nitpick is that the full path they include is likely not how this would manifest on a server. Everything after “src” would be the endpoint being scanned presumably: /Util/PHP/eval-stdin.php

Service Exploit #9: /hudson

With this, we run into another mess and rabbit hole. Radware summarizes this as “Hudson continuous integration tool – multiple vulnerabilities” and further describes Hudson as “a continuous integration tool written in Java, which runs in a servlet container, such as Apache Tomcat or the GlassFish application server. Over the years the project was replaced by Jenkins. The final release. 3.3.3 was on February 15, 2016. Today Hudson is no longer maintained and was announced as obsolete in February 2017.

Based on this description, this could be any one of at least 50 vulnerabilities going back to February, 2014, one of which does not have a CVE ID. 41 of these are in Jenkins software which is mentioned above.

Other Service Exploits

This is a curious conclusion to the “top 10” list, as it states “In addition to the new items that we covered in this list, we have also seen items that we already saw and covered in our previous blog Top 10 Web Service Exploits in 2019 such as /ctrlt/DeviceUpgrade_1, /TP/public/index.php and /nice%20ports%2C/Tri%6Eity.txt%2ebak.

That isn’t exactly a #10 on this list, rather a catch-all for “other stuff we saw including…“. The first listed tracks with VulnDB 170573 “Huawei HG532 Routers /ctrlt/DeviceUpgrade_1 NewStatusURL Element Remote Command Execution (Satori)” which is notable as it is used in Satori, a Mirai botnet variant.

The second tracks with VulnDB 194379 “ThinkPHP /public/index.php call_user_func_array() Function vars[1][] Parameter Remote Code Execution“. Note the different exploit path and we see it can actually be exploited via several endpoints according to analysis of the vulnerability by Knownsec 404 Team.

The third doesn’t immediately track with an entry in VulnDB. Radware gives us “/nice%20ports%2C/Tri%6Eity.txt%2ebak” which we can decode to a more friendly “/nice ports,/Trinity.txt.bak“. A quick Google for that request finds a blog from Dragos titled “Threat Hunting With Python Part 2: Detecting Nmap Behavior with Bro HTTP Logs” explaining this request:

The request for “/nice ports,/Trinity.txt.bak” comes from Nmap’s service detection routine testing how a server handles escape characters within a URI. The actual request is “GET /nice%20ports%2C/Tri%6Eity.txt%2ebak HTTP/1.0\r\n\r\n”.

So this isn’t an actual exploit, rather, it indicates that attackers are using the Nmap port scanner. This is a good reminder that “exploit scanning” doesn’t always cleanly map to a specific vulnerability.


Detecting exploitation is critical for every organization. Doesn’t matter if it is on-premises devices or a managed service-based detection. What is more critical is having comprehensive and timely vulnerability intelligence that can turn what you detect into actionable information. This is how you not only detect, but evaluate and remediate, assuming of course the vulnerability is known to the vendor or a mitigation can be enacted.

Sitting on Undisclosed Vulnerabilities (e.g. SolarWinds Stragglers)

The company SolarWinds is in the news, victims of an attack that compromised their Orion Platform software by inserting a backdoor into it, allowing for remote code execution. Like most big breaches, we hear the term “sophisticated” used for the attack. And like many breaches, we quickly learn that it might not have been so sophisticated after all. There is plenty of commentary on this and the wave of attribution experts are out in full force on Twitter. You can read about all of that elsewhere as I cover a different aspect of vulnerability disclosures here.

For anyone who has done penetration testing, they have found vulnerabilities of course. Since those tests are done under non-disclosure agreements (NDA), the vulnerabilities are reported to the customer. One long-standing problem with process is that a vulnerability found during that test may be in commercial off-the-shelf (COTS) software that affects many other organizations in the world. But that NDA often says you cannot disclose them elsewhere, including to the vendor. Even if it does, most penetration testing shops don’t have someone designated to handle coordinated disclosure with the vendor. When it does happen, it is often in the tester’s spare time or if the company uses security advisories for advertising, may task them to write it up.

For more than 25 years, this means that a lot of vulnerabilities are discovered in COTS that die in customer reports. The customers may sometimes report them to a vendor themselves looking for a fix. But surprisingly, that often does not happen. How do we know? Many testers have seen the exact same vulnerability during a test of the same customer a year or more after the original. There are times where a tester will disclose those vulnerabilities long after the fact, without coordinating with the vendor. This can happen after they leave the company they did the testing for or when they think sufficient time has passed.

I think we saw this yesterday with SolarWinds with the publication of CVE-2018-16243. First, while MITRE is not consistent about the assignment year, CVE is intended to use the year to denote when the vulnerability was discovered, not disclosed. A 2018 ID assigned to an issue that was published yesterday strongly suggests the researcher requested the ID back in 2018 but waited until now to publish. The exact date is likely 8/30/2018 per the disclosure itself. But looking at the disclosure, done via gist.github.com, we can see via the revisions that it was published 12/14/2020. So the researcher appears to have sat on these SolarWinds Database Performance Analyzer vulnerabilities for 837 days. Based on the disclosure, there was no coordination with the vendor and no fix currently available. On the upside, seven distinct XSS vulnerabilities were disclosed but the CVE only covers six of them.

Why now? Because SolarWinds was in the news albeit for a vulnerability in a different product (SolarWinds Orion Platform). Looking at prior vulnerability disclosures, it is easy to tell where the researcher works. A quick LinkedIn search verifies that bit of information and brings us to the fun question; did they find these SolarWinds vulnerabilities at their prior job, the downtime between jobs, or at Optiv? All three have interesting implications if you think about it. =)

Jumping back to the point, I will renew the call I have made in the past; penetration testing shops should use an NDA that allows them to report vulnerabilities in COTS to the vendors on behalf of the customer. They should manage the coordinated disclosure process and publish an advisory after a fix has been made available and they verified their customer has mitigated the vulnerabilities. Yes, it is a little extra work! Yes, it also is a value add to your customer, value to any organization that uses the software, and the advisories become advertising of sorts. That little extra work will go a long way for the greater good.

Not all CVEs are Created Equal. Or even valid…

[I wrote this early 2019 and it was scheduled for January 7 but it apparently did not actually publish and then got lost in my excessive drafts list. I touched it up this week to publish because the example that triggered this blog is old but the response is evergreen. Apologies for the long delay!]

I recently caught a Tweet from @NullCon offering 10 free conferences passes to NullconDasham, awarded to “InfoSec heroes who shared their hard work with the community & contributed to the @CVEnew database“.

I re-Tweeted with comment pointing out “There were at least 311 CVE assignments in 2018 alone, that were for issues that were *not* a vulnerability. I hope you are going to scrutinize the submissions.” Anant Shrivastava replied asking for some examples, and the next morning Mitja Kolsek suggested a blog would be beneficial. Here it is, as short and sweet as I can, which is never short in the world of Vulnerability Databases (VDBs) due to caveats.

Continue reading

Thoughts on 0-days and Risk in 2020

[Stupid WordPress. This was scheduled to publish Nov 23 but didn’t for some reason. Here it is, a bit late…]

On Friday, Maddie Stone from the Google P0 team Tweeted about the 0-day exploits her team tracks. As someone who checks that sheet weekly and tracks vulnerabilities, including ones ‘discovered in the wild’, this is a topic that is squarely in my tiny niche in the industry. Also, big fan of the P0 team!

I replied to her Tweet suggesting it come with a disclaimer that it didn’t represent “all” 0-days, rather they tracked high-end 0-day used primarily in “APT” attacks. Ben Hawkes, manager of the team, replied and agreed with that assessment. Before we proceed, let’s define 0-day real quick since the term is used for a variety of vulnerabilities, often incorrectly.

In this case, the context is a 0-day is a vulnerability that was actually found being exploited in the wild before there was public knowledge of it. In Risk Based Security’s VulnDB, we track that as “discovered in the wild“. Since VulnDB is comprehensive and our goal is to track every vulnerability, regardless of software or severity, we tend to aggregate a lot more than others. As of this post, we have over 78,000 vulnerabilities that aren’t found in CVE / NVD as a point of comparison. In my reply to Maddie I pointed out that we had seen 51 this year compared to their 22.

Next, Allen Householder replied to me asking a fun point, which is how many vulnerabilities did that really represent. Out of the 20,000+ vulnerabilities aggregated in 2020, we have 51 that are flagged as “discovered in the wild”. That represents only 0.25% of all vulnerabilities this year. One point I made previously is that Google’s team likely doesn’t care about a 0-day in the “Adning Advertising Plugin for WordPress” despite it being used to compromise WordPress blogs.

So with that number in mind, it goes back to the narrative that companies need to be scared of 0-days. They absolutely do! But… and this is the big qualifier that needs to come with that fear, is that perhaps they don’t need to be as afraid of 0-days as they do of already public vulnerabilities that they missed. With only 51 0-days in 2020, that means a vast majority of organizations simply aren’t likely to be targeted. Fully patching all known vulnerabilities that impact them should be priority one.

More to the point, vulnerabilities that have functional public exploits allowing anyone to trivially launch a viable attack are consistently a much bigger risk than the elusive 0-days. That is also one reminder of how often times CVSS falls short, if your vulnerability intelligence doesn’t provide Temporal scoring or exploit availability. Organizations making risk decisions only using the CVSS Base score are missing out on an important risk attribute.

I’ll end this blog with some arbitrary statistics around 0-days for fun! These are based on VulnDB data as of 11/21/2020. Note that metadata is less complete before 2012, which includes ‘discovered in the wild’ classification.

  • 241,690 vulnerabilities, only 641 are 0days (0.27%)
  • 14 are in Google products: Chrome (5), V8 (3), Android (6)
  • 146 are in Microsoft products: Windows (63), IE (36)
  • 13 are in Apple products
  • 7 are in Oracle products: Java (4)
  • 62 are in Adobe products: Flash (38), Reader (14)
  • 18 are in security products 😞
  • The oldest is from 1975 in RSTS/E! Yes, for real.
  • The oldest you likely recognize is Sendmail in November, 1983

The Hacker Jeopardy That Never Was

Many years ago, at early DEF CONs before 2000, I became a critic of Hacker Jeopardy after some of the questions had wrong answers. The host had written the questions and answers but got some wrong. The next year I offered to sanity check them before the game and did so, finding a few errors shortly before the game started. I think this happened a year after that but my memory is fuzzy as to how many years I helped. At some point I offered to help write questions well in advance of the next DEF CON and began scribbling ideas in a notebook. I found that notebook recently!

Below were to be the proposed topics and questions in order of difficulty. I have not included a few questions which would have been acceptable to most attendees back then, but shouldn’t have been in hindsight. One of the questions revolved around ‘open secrets’ of two individuals in the scene, one being John Draper and the other continuing to be an open secret to this day.

After a recent DEF CON which had a Hacker Jeopardy that had every team miss which port Telnet is on [23], I wonder how teams would do with these. Some may be subjective, but they had more widely-known backstories at the time.

Errata

  • This charlatan is best known for her delusions of grandeur, Erik Bloodaxe reading her mail, the FBI harassing her, and more. [Carolyn Meinel]
  • PGP is a lost concept to this charlatan. [Winn Schwartau]
  • This well-trained monkey/charlatan hacked a bank once. [Ira Winkler]
  • This charlatan can help you learn the SECRETS of hacking a public library or BBS. [Knightmare]
  • This charllatan is master of using ‘grep’ for his IDS at NASA! [Dan Ridge or ‘B-grep’ or ‘wizkid’]

Which DEF CON

This first answer on my list strikes me as wrong. My own memory today says only ~ 200 showed up to DEF CON 2, but now I wonder if it was really ~ 400, which would explain an answer of ~ 300 showing to DEF CON 1. But conventional wisdom and our poor memories often cite the first one only having ~ 100 there. Anyone have a more definitive memory?

  • Only 300 people showed to this DEF CON [1]
  • Which two hackers were thrown out of the Aladdin at which DEF CON? [Pete Shipley / Voyager @ 5]
  • The Sahara was serving minors Heineken beer at which DEF CON? [2]

W’ere here to help…

  • We are hackers who will be glad to narc you for teenpron.gif! [EHAP or Ethical Hackers Against Porn]
  • We are hackers who will be glad to get you legal counsel like the other 0 we have helped. [HDF or Hackers Defense Foundation]
  • Spending a quarter million to prove what everyone knew by building “DeepCrack” is the only thing we’ve done in years. [EFF]
  • We’ll be glad to repost your advisories six months after you do! [CERT]
  • Pay us thousands, and our 17 year veterans will babble … err teach you to hack Japanese banks. [se7en]

DX3BH!

  • What does RSA stand for? [Rivest, Shamir, Adleman]
  • Win95 SSH supports what flavors of encryption? [Idea, 3DES, Blowfish]
  • Name one ITAR loophole [printing or missile]
  • What crypto engine is unix crypt() based on? [Enigma]

Everything under the Sun

  • Sun was derived from what flavor of Unix, while Solaris hails from which? [BSD vs SysV]
  • What is the default debugger installed with Solaris? [adb]
  • How many returns does it take t overflow AND exploit a vulnerable binary on the sparc architecture? [2]

Ancient Exploits

(I only had notes for 4 questions, nothing written out)

  • SunOS 4.0.3
  • Convex
  • Unicos 7.x
  • BBS

Fucking Unix

The idea for this was Unix commands that were also commonly joked about euphemisms for sexual activity. There were many, many more back in the day but I only ended up with three questions in my notes for some reason.

  • Foreplay as ‘stinky pinky’ [finger]
  • This function might lead to child processes [fork()]
  • These two commands make 69 [head + tail]

Ultimately these were never used I don’t believe, and as I recall, the host and question writer for Hacker Jeopardy at the time said ‘yes’ to collaborating on questions in advance of the next convention, but did not follow-through at all so the idea died off.

I don’t recall what “8.6” referred to for the answer to the first question under ‘Everything under the sun’, so I didn’t include it above.

More authorities, more CVEs; Oh, and more commentary.

On November 10, TechBeacon published a great article by Rob Lemos titled “More authorities, more CVEs: What it means for app sec teams” in which I was quoted, along with several other people.

Like many articles of this nature, those who provide input often will talk for as long as half an hour and ultimately get a couple lines quoted. We do it to provide background and context on the topic as well as have an open discussion on vulnerability trends. That means there are ‘outtake’ opinions and facts, as well as our potential reaction to other parts of the article that did not include our input. So this blog just covers some of my random observations to compliment the article.

Until 2016, more than 80% of software security issues assigned a CVE identifier belonged to only 10 classes, or weaknesses, as classified by their Common Weakness Enumeration (CWE) category. But in 2019, the top 10 weaknesses only accounted for 59% of reported vulnerabilities.

The Common Weakness Enumeration (CWE) index is interesting to me and I wonder if it has gotten so big to degrade its value. Consider that there are now 891 CWE identifiers as of August 20 in version 4.2 of the framework. Per the article, only 10 of them account for 59% of vulnerabilities which will no doubt include XSS, SQLi, and CSRF as examples. That makes me wonder the value of abstracting so much as it means that hundreds of those CWEs will represent a handful of vulnerabilities at most.

Digging into the 2,298 page PDF documenting version 4.2, you can jump toward the end of the CWE list and see that several have been created but have no “Observed Examples”. In fact, searching for that phrase only yields 397 hits. Does that mean that out of 891 CWE IDs representing weaknesses, that MITRE has only come up with 397 that match known vulnerabilities? I certainly expect otherwise and hope this is just documentation shortcoming as I feel that every CWE ID should be linked to a concrete real-world example. I’d love to see

I’d love to see a simple breakdown of the top 100 CWE along with how many vulnerabilities are associated with them (via NVD, since MITRE doesn’t actually apply CWE to entries) and what percentage of the overall vulnerabilities that represents. It might be very telling just how useful CWE is and if the project is being pushed too heavily from an academic standpoint. Before you judge that comment, let me know how useful this CWE report from MITRE is, and make sure you load it in Chrome.


It’s an open question whether the addition of coding repositories will lead to another expansion in the number of vulnerabilities.

I don’t think that is an open question at all. I think the number of vulnerabilities will go up as a result of more coding repositories becoming a CNA. But that isn’t really the issue here. Instead, the real questions should be centered around what quality of CVE entries they will provide, if they will adhere to CNA standards, and if MITRE will enforce CNA policy on them.

Based on history, the short answers to those questions are: quality will go down, no, and no. As soon as MITRE provides a breakdown of how many IDs were published by each CNA it is difficult to know. Speaking of, why hasn’t MITRE published such statistics? Rhetorical question, apologies.


Open source vulnerabilities: Do you know what’s in your software?

I, along with many others, can’t stress this enough! Please make sure you understand what third-party software your developers are using. This affects your organizations from both a vulnerability standpoint, as well as legal accountability. Using a third-party library against its license could open you up to some hardships.

The only reason I quoted this section is because I just read an article in the latest Wired that mentions Bootstrap is thought to be used on nearly 20% of all web sites across the Internet. That is incredible.


Patch Tuesday is becoming a bottleneck

There is a lot more that can be said on this topic. It reminds me of a 2015 blog I wrote that actually goes back farther to 2007 where this problem was predicted long before the nightmare IT teams experience once a month. It’s only going to get worse as more vendors jump on this patch schedule and the only ones who will suffer are their paying customers. But hey, let’s let them keep using the term “responsible” disclosure too.


But exploitability alone doesn’t solve the problem—three quarters of the 17,300 vulnerabilities identified ranked CVSS exploitability rating of 8.0 or higher.

I’m not sure a more perfect example of why CVSS has become worthless exists. On its own, especially using the Base score only, is it really helpful that so many vulnerabilities are ‘High Risk‘? This is also a good reminder of another blog I have been meaning to write for a while that outlines the distribution of CVSSv2 versus CVSSv3 and how it impacts scoring. With a couple charts you will get a nice visual of just how poorly thought out some of the framework was. Of course, this has already been evaluated by others years back as well.


Finally, because I don’t hold the copyright to the picture used in the TechBeacon article header, I offer my version:

Picture of me by D2d in November, 2018 at Tomayo, in Denver, CO.

Vulnerability Counts Are a Moving Target

At the end of each year, we see articles covering how many vulnerabilities were disclosed the prior year. Because the articles are written about the same time of year, it gives a fairly good initial comparison from year to year; at least, on the surface. This is the foundation of statements such asSecurity vulnerabilities in critical infrastructure up 600%”. My company, Risk Based Security, even includes that general type of observation in our vulnerability reports, with caveats. These sensationalized counts and figures are also often used to make claims that one product is more or less secure than another, when the vulnerability counts cannot typically be used for such claims as they are built on severely incomplete data. In reality, we must remember that such numbers are only a snapshot in time and serve as a quick comparison between years, not much more.

Before we get to the “moving target” topic, we need to cover a bit of background on how all this happens.

First, consider that even with a large team doing vulnerability aggregation, there is a limit to the number of sources that can be monitored. While a team might monitor over 4,000 sources on a daily to weekly basis, we know there are a lot more out there. As new researchers create their blogs, older vendors finally create advisory pages, and new vendors pop up, the new sources are growing at an incredible rate. Additionally, consider that there are over a million results for “site:github.com changelog.md” (not to mention variations like “release_notes” or “changelog.txt” and similar) that could potentially host a trove of vague vulnerability mentions. Even more daunting, back in 2010 GitHub was hosting 1 million repositories and now they are over 100 million. That means there are an overwhelming number of bug trackers, pull requests, and over a billion commits on a single site. Any company that claims to monitor all of that or “millions” or sources? Do your due diligence and be ready to walk away from them.

Second, due to available resources, vulnerability aggregation teams have to prioritize their activity. This is usually done by vendor, product, and the time frame where higher deployment vendors and products get the most attention. With “time frame”, emphasis is placed on the more recent vulnerabilities as they are most likely to be relevant to organizations. Moving past that, a vulnerability intelligence (VI) provider must be working with clients to learn what resources they use in their organization, as it allows them to prioritize and ensure that they are covering exactly what is deployed first and foremost. After all that, as time permits, they have to come up with new ways to expand source coverage without compromising quality or speed.

With that in mind, consider a vendor that finally publishes a changelog or makes their bug tracker open for everyone. While a VI team should love to go through such resources as far back as possible, they have to limit themselves to vulnerabilities for the current year, and then some amount of time farther back in case clients are using older versions (especially for third-party dependencies). Time permitting, the team should then go back even further to dig for older and more obscure vulnerabilities. While these may or may not immediately benefit clients based on the software they are running, it does contribute directly to the vulnerability history of a given product or vendor. This is invaluable in determining the “cost of ownership” for a product and is vital to making a decision between multiple vendors offering the same type of solutions. With all of that data, it is trivial for a VI provider to provide a quick and easy-to-understand picture of that cost.

Even with very limited time to dig that far back into sources, the impact can still be seen clearly. In January of 2013, Risk Based Security’s VulnDB team had aggregated 8,822 vulnerabilities for the 2012 calendar year, and CVE covered only 4,765 of them (54%). Compared to the prior year (7,911 in 2011), we could say that disclosures increased around 10%. The next question we must ask is if those numbers aged well and hold true today.

Looking at VulnDB now, there were 10,856 vulnerabilities disclosed in 2012. So in the past eight years, the team has managed to find an additional 2,034 vulnerabilities disclosed that year. That means comparing 2012’s updated 10,856 count with the older 7,911 count for 2011, the percent increase was closer to 28%. But wait, we can no longer use the 7,911 count for 2011 either, since that too is a moving target! Ultimately, as previously stated, these disclosure numbers are only good as a snapshot in time. Depending when you perform the count, you may find wildly varying results that could heavily bias any conclusions you try to make. Do the people writing the statistics you read and cite disclaim that?

In January of 2013, I started taking counts every month for how many vulnerabilities VulnDB aggregated for the 2012 calendar year. Almost eight years later, and this blog and chart shows just how much that number can change. As with all vulnerability statistics, please make sure you fully understand what they really mean and disclaim as needed!

Viewing 2012 disclosures; chart shows 7k / 11k as min / max values.

With this visual we can see that in the years after 2012, vulnerability aggregation continued to rise considerably. Over time that growth tapers off as the team simply didn’t have time to keep digging that far back into changelogs and bug trackers looking for more vulnerabilities that were less beneficial to organizations.

The tl;dr takeaways:

  • The number of vulnerabilities disclosed in a given year is static, but the VI teams that aggregate the information won’t find them all that year.
  • Vulnerabilities by year, as reported, will slowly climb over time as additional aggregation work is performed.
  • While that newly aggregated data may be “old” as far as what software is being used within an organization, it still contributes to better metadata and product/vendor evaluation (“cost of ownership”)

“The History of CVE” and A Couple of Objections

I just read “The History of Common Vulnerabilities and Exposures (CVE)” by Ary Widdes from Tripwire and found it to be a great summary of the 20+ years of the program. I say that as an outspoken CVE and MITRE critic even! I do have a couple of objections however, with the conclusion, and then a fun bounty!

Widdes concludes the history by saying:

A lot has changed in the 21 years since the CVE List’s inception – both in terms of technology and vulnerabilities. Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage. It’s also possible that someone else would have created a service similar to the CVE List. Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

There’s a lot to unpack here so I will take it one sentence at a time, starting with the second.

“Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage.”

No, there is no “possible” here. That is a simple reality with an important caveat. The reality is that teams of all types still use multiple tools from multiple vendors to do their job. The caveat, and more to the point of that sentence, is that CVE doesn’t offer “complete coverage” and many of the vulnerability scanners only cover a third of the issues in CVE for various reasons. Even using a combination of firewalls, vulnerability scanners, intrusion detection/prevention, audits, and a slew of other tools, organizations are likely seeing half of what CVE has to offer at best. Widdes’ conclusion here gives undue credit to CVE and the state of vulnerability coverage it offers.

It’s also possible that someone else would have created a service similar to the CVE List.

This is where the vulnerability historian in me wants to rage a bit. This statement is unequivocally false for the simple reason that vulnerability databases existed before CVE, both free (e.g. X-Force) and commercial (e.g. RSI), in 1997 alone [1]. The first vulnerability database was created in 1973, specific to Multics, but also when there weren’t that many other systems to catalog bugs or vulnerabilities in. In 1983 we saw the Mt Xinu Bug List and in 1985 Matt Bishop’s List of UNIX Holes, both of which were more comprehensive than one platform. If we consider a vulnerability database implemented via product, we had ISS, SATAN, Ballista, and Nessus between 1995 and the creation of CVE in 1999. Many of the hackers turned security professionals may fondly remember Fyodor’s Exploit World (1996 – 1998) from both aspects of their lives. Those same folks probably also remember Packet Storm (1998) which is still running today.

Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

This, unfortunately, is true. I say unfortunately because of my long-standing criticisms of CVE over the past decade, but won’t go into here.

Bug(s) Bounty:

If there is anyone at MITRE open to outright bribery, including all-you-can-eat sushi dinners, I will pay a bounty to get my hands on that list of 8,400 submissions! While I know there are likely a lot of duplicates, the vulnerability historian in me would love to audit that data to see if MITRE decided to skip any that would be considered vulnerabilities by today’s standards, or where someone else back then had more knowledge of a vulnerability than was submitted. That data is over twenty years old and was solicited, processed, and partially published with U.S. taxpayer funded money. There’s no reason not to make it public. =)

[1] The Repent Security Inc. (RSI) database existed in 1997 but may not have been offered as a commercial product until 1998.

Disclosure Repair Timelines?

For those in InfoSec, you have probably seen a vulnerability disclosure timeline. Part of that often includes the researcher’s interaction with the vendor including the vulnerability being fixed. After the issue is disclosed, the story typically ends there. Every so often, work needs to be done after that to ‘repair’ part of the disclosure.

For the last year or more I have found myself having to follow-up on more disclosures, specifically because someone on Twitter has posted using an incorrect CVE ID associated with the vulnerability. One of the cornerstones of a CVE assignment is to give it a unique identifier that makes that vulnerability distinct from any others that may be similar. Using the incorrect CVE ID can actually cause a lot of headache for threat intelligence folks that monitor for vulnerability disclosures.

Often times I send a message and within a day the errant CVE ID is fixed. The errors tend to be nothing more than a typo or transposition issue. When fixed quickly and not further indexed by search engines and cited or included by news aggregator sites, the problem is over. Once the errant ID is in several reputable (or somewhat reputable) sources, it is more prone to be quoted in additional blogs and spread from there. Catching and fixing these errors needs to happen quickly, but unfortunately MITRE, the organization responsible for CVE, does nothing in this regards.

The past two weeks, I ran into what is probably the worst case as far as time and effort required to fix a single incorrect CVE. I thought I would share what the timeline looks like as this is not something anyone typically tracks that I am aware of, myself included. But it shows that even after a disclosure more work may need to be done to ensure clarity in it. I’m withholding names because while this time around was difficult, the journalist and publication has quickly fixed other typos in the past. My goal is to show that timely corrections are what is best for the community.

4/30 – Article published citing four CVE IDs, one incorrect.
4/30 – Ping publication/journalist on Twitter
5/1 – Bump thread
5/8 – Bump thread
5/13 – Tweet again asking for a correction
5/14 – Submit site feedback via two different forms
5/14 – Tweet frustration at publication
5/15 – Publication replied to form, didn’t seem to fully understand the point
5/19 – Sent a DM to the author of the article pointing to original Tweet
5/21 – Author replied saying they will fix it. Article amended to fix and clarify the error.

Twenty one days to fix is rough. Publications and journalists; please understand that a CVE ID is important to get right. If you have any questions about CVE, how it works, or the importance, please feel free to reach out. I am happy to take the time to help you.