Redscan’s Curious Comments About Vulnerabilities

As a connoisseur of vulnerability disclosures and avid vulnerability collector, I am always interested in analysis of the disclosure landscape. That typically comes in the form of reports that analyze a data set (e.g. CVE/NVD) and draw conclusions. This seems straight-forward but it isn’t. I have written about the varied problems with such analysis many times in the past and yet, companies that don’t operate in the world of vulnerability databases still decide to play in our mud puddle. This time is the company Redscan, who I don’t think I had heard of, doing analysis on NVD data for 2020. Risk Based Security wrote a commentary on their analysis, to which I contributed, but I wanted to keep the party going over here with a few more personal comments. Just my opinions here, as a more outspoken critic on the topic, and where I break from the day job.

I am going to focus on one of my favorite topics; vulnerability tourists. People that may be in the realm of Information Security, but don’t specifically operate day-to-day in the world of vulnerability disclosures, and more specifically to me, vulnerability databases (VDBs). For this blog, I am just going to focus on a few select quotes that made me double-take. Read on after waving to Tourist Lazlo!

“The NVD tracks CVEs logged by NIST since 1988, although different iterations of the NVD account for some variation when comparing like-for-like results over time.”

There’s a lot to unpack here, most of it wrong. First, the NVD doesn’t track anything; they are spoon-fed that data from MITRE, who manages the CVE project. Second, NIST didn’t even create NVD until over five years after CVE started. Third, CVE didn’t track vulnerabilities “since 1988”; they cherry-picked some disclosures from before 1999, when they started, and why CVE IDs start with ‘1999’. Fourth, there was only one different iteration of NVD, that was their ICAT “CVE Metabase” that ran the first year of CVE basically. According to Peter Mell, who created it, said that after starting as its own vulnerability website, “ICAT had become an archival tool for CVE standard vulnerabilities and was only updated every three or four weeks”. Then in 2005 the site relaunched with a new focus and timely updates from CVE. Despite this quote, later in their report they produce a chart that tries to show an even comparison from 1988 to 2020 despite saying it went through iterations and despite not understanding CVSS.

“The growth is also likely attributable to an increase in the number of CVE Numbering Authorities (CNAs) – of which there are now more than 150 worldwide with the power to create and publish CVEs.”

The growth in disclosures aggregated by CVE is a lot more complicated than that, and the increase in CNAs I doubt is a big factor. Of course, they say this and don’t cite any evidence despite CVE now showing who the assigning CNA was (e.g. CVE-2020-2000 is Palo Alto Networks). The data is there if you want to make that analysis but it isn’t that easy since it isn’t included in the NVD exports. That means it requires some real work scraping the CVE website since they don’t include it in their exports either. Making claims without backing them up when the data is public and might prove your argument is not good.

“Again, this is a number that will concern security teams, since zero interaction vulnerabilities are famously difficult to detect and have the potential to cause significant damage.”

This makes me think that Redscan should invent a wall, perhaps made of fire, that could detect and prevent these attacks! Or maybe a system that is designed to detect intrusions! Or even one that can prevent intrusions! This quote is one that is truly baffling because it doesn’t really come with an explanation as to what they mean, and I hope they mean something far different than what this sounds like. I hope this isn’t a fear tactic to make readers think that their managed detection service is needed. Quite the opposite; anyone who says the above probably should not be trusted to do your attack detection.

This chart heading is one of many signs that Redscan doesn’t understand CVSS at all. For a “worst of the worst” vulnerability they got several attributes right but end up with “Confidentiality [High]”. The vulnerability they describe would only be CVSSv2 7.8 (AV:N/AC:L/Au:N/C:C/I:N/A:N) and CVSSv3.1 7.5 (AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). That is not the worst. If it ‘highly’ impacts confidentiality, integrity, and availability then that becomes the worst of the worst, becoming CVSSv2 10.0 and CVSSv3.1 9.8 or 10.0 depending on scope. It’s hard to understand how a security company gets this wrong until you read a bit further where they say they “selected the very ‘worst’ option for every available metric.” My gut tells me they didn’t realize you could toggle ‘High’ for more than one impact and confidentiality is the first listed.

“It is also important to note that these numbers may have been artificially reduced. Tech giants such as Google and Microsoft have to do a lot to maintain their products and services day-to-day. It is common for them to discover vulnerabilities that are not being exploited in the wild and release a quick patch instead of assigning a CVE. This may account for fewer CVEs with a network attack vector in recent years.”

This is where general vulnerability tourism comes in as there is a lot wrong here. Even if you don’t run a VDB you should be passingly familiar with Microsoft advisories, as an example. Ever notice how they don’t have an advisory with a low severity rating? That’s because they don’t publish them. Their advisories only cover vulnerabilities at a certain threshold of risk. So that means that the statement above is partially right, but for the wrong reason. It isn’t about assigning a CVE, it is about not even publishing the vulnerability in the first place. Because they only release advisories for more serious issues, it actually skews their numbers to include more remote vulnerabilities, not less, primarily on the back of “remote” issues that require user interaction such as browser issues or file parsing vulnerabilities in Office.

This quote also suggests that exploitation in the wild is a bar for assigning a CVE, when it absolutely is not. It might also be a surprise to a company like Redscan, but there are vulnerabilities that are disclosed that never receive a CVE ID.

“Smart devices designed for the mass market often contain a worrying number of vulnerabilities due to manufacturer oversight. Firmware within devices is often used by multiple vendors, meaning that any vulnerabilities in this software has the potential to result in lots of CVEs.”

Wrong again, sometimes. If it is known to be the same firmware used in multiple devices, it gets one CVE ID. The only time there are additional IDs assigned is when multiple disclosures don’t positively ID the root cause. When three disclosures attribute the same vulnerability to three different products, it stands to reason there will be three IDs. But it isn’t how CVE is designed because it artificially inflates numbers, and that is the game of others.

“The prevalence of low complexity vulnerabilities in recent years means that sophisticated adversaries do not need to ‘burn’ their high complexity zero days on their targets and have the luxury of saving them for future attacks instead.”
-vs-
“It is also encouraging that the proportion of vulnerabilities requiring high-level privileges has been on the increase since 2016. This trend means that cybercriminals need to work harder to conduct their attacks.”

So which is it? When providing buzz-quote conclusions such as these, that are designed to support the data analysis, they shouldn’t contradict each other. This goes back to what I have been saying for a long time; vulnerability statistics need qualifications, caveats, and explanations.

“Just because a vulnerability is listed in the NVD as hard to exploit doesn’t mean that attackers aren’t developing PoC code to exploit it. The key is to keep up with what’s happening in the threat landscape and respond accordingly.”

I’ll end here since this is a glowing endorsement for why vulnerability intelligence has to be more evolved than what CVE and NVD are offering. Part of the CVSS specifications include Temporal scoring and one of those attributes is Exploit Code Maturity. This is designed to specifically address the problem above; that knowing the capability of potential attackers matters. With over 21,000 vulnerabilities disclosed last year, organizations are finding that just patching based on the CVSSv3 base score isn’t enough. Sure, you patch the 10.0 / 9.8 since those are truly the worst-of-the-worst, and you patch the 9.3 / 8.8 since any random email might carry a payload. Then what? If all things are equal between vulnerabilities that impact your organization you should look to see if a patch is available (also covered by Temporal score) and if an exploit is available.

Numeric scores are not enough, you have to understand the context behind them. That CVSSv2 remote information disclosure that partially affects confidentiality by disclosing an admin password is only a 5.0. Score it under CVSSv3 and you are looking at a 9.8 because it immediately leads to privilege escalation which is factored in under that system. Heartbleed was a CVSSv2 5.0 with a functional exploit and available patch; look what hell that brought upon us. If you aren’t getting that type of metadata, reconsider your choice of vulnerability intelligence.

Search Speak for Automaton

Alternate titles for this blog could be “Doodle Transition for Machina” perhaps! For at least a decade I have thought about just such an application and today I have Google Translate for Android. Load, aim, and it will process the text and translate on screen for you. Given the state of technology you would think it would be amazing by now, and it sometimes is.

The success largely depends on the language and that can also be seen in using translate.google.com, where some languages will translate fairly cleanly and others are very rough. One language I have to translate frequently is Chinese (simplified) and it is problematic for many things including company names and technical terms. With that in mind, I would expect it to translate with the same issues via the Google Translate app, and more specifically, do so consistently.

Since I am writing this, you know what’s coming…

This is the result of holding the phone up to a mail label from Japan. That’s all! Just moving the phone ever so slightly by tilting it or moving it half an inch closer / farther will make it change the translation. I think it finally got it a bit correct on that last one since the envelope didn’t contain anything living.

Hopefully the translation technology from Google will advance more quickly on Asian languages. Until then, I am just glad I didn’t get any “Sunrise Holy Poop” in that envelope.

Twitter, Companies, and your Complaints

The rise of social media has been interesting to say the least. Many on twitter have found it to give them a type of power as they can voice their complaints directly to a company that has wronged them. Everything from bad customer service, bad prices, minor inconvenience, or even perceived slights that likely never happened as described.

This ability has given rise to social media teams at these companies that are often extensions of the customer support teams that traditionally handle phone and email based contact. Since the complaint isn’t a direct communication between the offended and offender, companies have figured out that it behooves them to control the narrative as much as possible. Since the original Tweet(s) have outlined a bad experience they must try to head off any additional commentary be it from the offended to friends of theirs sharing their own negative experiences or even random users that see a RT or search for those sharing negative stories.

Without fail, the offending company will reply and immediately ask you to take it to direct messages (DMs) to control the narrative. They show they are quick and eager to resolve your issue! After that they only need to provide a base level of customer service and hope that satisfies you. It is interesting to note that they will do this by asking you to send your name and specific information to assist you, even if your complaint isn’t specific to you. Don’t let them do this.

If your complaint is generic and not specific to your account or personal details, don’t go to DMs with them. Have the conversation publicly so everyone can see it and those searching down the road can find it. If you do take it to DMs and they don’t resolve it? Take them to task, again. Keep doing it until they make things right or ignore you like OptumRX did with me. Apparently you can only call them out for dreadful customer service and a web portal written by seven year-olds so many times before they give up trying to get you to go to DMs and away from the public eye.

How Many Trees Are You Celebrating @arborday?

The Arbor Day Foundation is a 501c(3) nonprofit organization founded in 1972 that seeks to “inspire people to plant, nurture, and celebrate trees“.

I received a “Colorado Tree Survey” from them today, part of what is a never-ending stream of snail-mail spam that I have written about before. For this envelope, the thing that caught my attention was the weight of the envelope.

To say that it was heavier than other junk mail would be an understatement. In fact, it was well over 5x heavier than most of my junk mail including the ones that send small pads of paper and mailing labels.

That’s right, it came in over three pounds. For Arbor to presumably send these out to at least hundreds of thousands of people in Colorado. It’s fair to say that took quite a few trees to produce not to mention the general carbon emissions required to produce and distribute. This seems to be at odds with their general mission purpose especially in the age of email.

Given that 23.4% of their income is not spent on program expenses it is discouraging to say the least. To put it into better perspective, over six million dollars went to administrative expenses and their CEO Matt Harris made $336,445 in 2019. Considering that $961,603 of their income came from government grants that year it is really frustrating to see nonprofit entities spend money on such salaries and waste money on killing trees to conduct surveys about celebrating trees.

If I print this blog out and mail it back in place of the survey, I wonder if they would appreciate the irony.

“The History of CVE” and A Couple of Objections

I just read “The History of Common Vulnerabilities and Exposures (CVE)” by Ary Widdes from Tripwire and found it to be a great summary of the 20+ years of the program. I say that as an outspoken CVE and MITRE critic even! I do have a couple of objections however, with the conclusion, and then a fun bounty!

Widdes concludes the history by saying:

A lot has changed in the 21 years since the CVE List’s inception – both in terms of technology and vulnerabilities. Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage. It’s also possible that someone else would have created a service similar to the CVE List. Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

There’s a lot to unpack here so I will take it one sentence at a time, starting with the second.

“Without the CVE List, it’s possible that security professionals would still be using multiple tools from multiple vendors just to ensure complete coverage.”

No, there is no “possible” here. That is a simple reality with an important caveat. The reality is that teams of all types still use multiple tools from multiple vendors to do their job. The caveat, and more to the point of that sentence, is that CVE doesn’t offer “complete coverage” and many of the vulnerability scanners only cover a third of the issues in CVE for various reasons. Even using a combination of firewalls, vulnerability scanners, intrusion detection/prevention, audits, and a slew of other tools, organizations are likely seeing half of what CVE has to offer at best. Widdes’ conclusion here gives undue credit to CVE and the state of vulnerability coverage it offers.

It’s also possible that someone else would have created a service similar to the CVE List.

This is where the vulnerability historian in me wants to rage a bit. This statement is unequivocally false for the simple reason that vulnerability databases existed before CVE, both free (e.g. X-Force) and commercial (e.g. RSI), in 1997 alone [1]. The first vulnerability database was created in 1973, specific to Multics, but also when there weren’t that many other systems to catalog bugs or vulnerabilities in. In 1983 we saw the Mt Xinu Bug List and in 1985 Matt Bishop’s List of UNIX Holes, both of which were more comprehensive than one platform. If we consider a vulnerability database implemented via product, we had ISS, SATAN, Ballista, and Nessus between 1995 and the creation of CVE in 1999. Many of the hackers turned security professionals may fondly remember Fyodor’s Exploit World (1996 – 1998) from both aspects of their lives. Those same folks probably also remember Packet Storm (1998) which is still running today.

Either way, from idea to whitepaper to database, the CVE List has become a core part of vulnerability and patch management.

This, unfortunately, is true. I say unfortunately because of my long-standing criticisms of CVE over the past decade, but won’t go into here.

Bug(s) Bounty:

If there is anyone at MITRE open to outright bribery, including all-you-can-eat sushi dinners, I will pay a bounty to get my hands on that list of 8,400 submissions! While I know there are likely a lot of duplicates, the vulnerability historian in me would love to audit that data to see if MITRE decided to skip any that would be considered vulnerabilities by today’s standards, or where someone else back then had more knowledge of a vulnerability than was submitted. That data is over twenty years old and was solicited, processed, and partially published with U.S. taxpayer funded money. There’s no reason not to make it public. =)

[1] The Repent Security Inc. (RSI) database existed in 1997 but may not have been offered as a commercial product until 1998.

Why @anacondainc Doesn’t Fully Understand CVEs

It’s worrisome that in 2020 we still have people in influential technical roles that don’t understand CVE. A friend told me earlier this year he was in a meeting where someone said that CVE IDs are assigned in order, so CVE-2020-9500 meant there were 9500 vulns in 2020 so far. Of course that is not how it works and a dangerous understanding of CVE.

I ran across an article written by Nick Malkiewicz of Anaconda titled “Why Understanding CVEs Is Critical for Data Scientists“. This article has several bits that show a lack of understanding of what CVE is. One of the biggest is equivocating a CVE with a vulnerability. Yes, many vulnerabilities directly map to a single CVE identifier, but a CVE is the identifier not the vulnerability. Additionally, sometimes one vulnerability can track with multiple CVE IDs, or one CVE ID can track to multiple vulnerabilities. So lines like the following are concerning:

When someone finds a CVE, they report it to a CVE Numbering Authority (CNA).

When someone finds a vulnerability, they report it to MITRE or a vendor, who may be a CNA but more often not one. That vendor can then ask MITRE for an ID via a web form.

CNAs assign identification numbers to CVEs and list them in publicly accessible databases.

A CNA is required to inform MITRE after a CVE-assigned vulnerability has been disclosed. That is actually a fairly recent rule, implemented in the last few years. For most of CVE’s history there was no requirement or specific communication channel for a CNA to notify MITRE of this. That was one of many failings of the CVE ecosystem and directly led to companies being breached, as they relied on CVE to be ‘complete’ and timely.

Each vulnerability listed in a CVE database has a score from .1 to 10, 10 being the highest risk level. These scores are based on exploitability, impact, remediation level, report confidence, and other qualities.

Technically, not even the first line is true as NVD can score a vulnerability as 0.0, meaning it is not a vulnerability and poses no risk. This occurs when a researcher or vendor disclose a vulnerability but don’t fully understand the issue or the subsequent impact. This happens hundreds of times a year although many are not included in NVD. The second sentence from Anaconda is also incorrect as NVD only scores CVSS Base scores. The exploitability, remediation level, and report confidence are part of Temporal scores and not included. You can see an example with CVE-2020-2800 published by Oracle and given a CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N score by both Oracle and NVD. This misunderstanding of NVD CVSS scoring is more baffling as Anaconda links to the same FIRST CVSS document I do in this paragraph.

Anaconda goes on talking about how there are other factors at play including looking at the history of a package, how fast vendors respond, and more. This is great advice and critical for companies facing tens of thousands of vulnerabilities a year. Unfortunately, they slide into the “more lipstick on that pig” problem:

The good news is, there are tools that automate the CVE monitoring process.

This is true. But, more ways to manipulate bad data still leaves you with bad data. In addition to CVE missing several thousand vulnerabilities a year, their push for quantity in the last few years have led to a serious hit on quality. There are some CVE IDs that have descriptions missing critical information like the vendor, affected version, or impact. All the data wrangling and fancy analysis of that data is still based on bad or incomplete information. All the lipstick on that pig still makes it a pig.

Finally, I will quote on other line from their blog that is curious:

Hacking open-source software also has a bigger payoff because many more people use it.

I understand and appreciate the sentiment, and finding a vulnerability in a library like OpenSSL obviously has a huge impact. However, that and a couple dozen libraries are still the outliers in the bigger picture. Compare your vulnerabilities like EternalBlue to a standard open source library vulnerability and they are not even close as far as how “many more people use it”.

Disclosure Repair Timelines?

For those in InfoSec, you have probably seen a vulnerability disclosure timeline. Part of that often includes the researcher’s interaction with the vendor including the vulnerability being fixed. After the issue is disclosed, the story typically ends there. Every so often, work needs to be done after that to ‘repair’ part of the disclosure.

For the last year or more I have found myself having to follow-up on more disclosures, specifically because someone on Twitter has posted using an incorrect CVE ID associated with the vulnerability. One of the cornerstones of a CVE assignment is to give it a unique identifier that makes that vulnerability distinct from any others that may be similar. Using the incorrect CVE ID can actually cause a lot of headache for threat intelligence folks that monitor for vulnerability disclosures.

Often times I send a message and within a day the errant CVE ID is fixed. The errors tend to be nothing more than a typo or transposition issue. When fixed quickly and not further indexed by search engines and cited or included by news aggregator sites, the problem is over. Once the errant ID is in several reputable (or somewhat reputable) sources, it is more prone to be quoted in additional blogs and spread from there. Catching and fixing these errors needs to happen quickly, but unfortunately MITRE, the organization responsible for CVE, does nothing in this regards.

The past two weeks, I ran into what is probably the worst case as far as time and effort required to fix a single incorrect CVE. I thought I would share what the timeline looks like as this is not something anyone typically tracks that I am aware of, myself included. But it shows that even after a disclosure more work may need to be done to ensure clarity in it. I’m withholding names because while this time around was difficult, the journalist and publication has quickly fixed other typos in the past. My goal is to show that timely corrections are what is best for the community.

4/30 – Article published citing four CVE IDs, one incorrect.
4/30 – Ping publication/journalist on Twitter
5/1 – Bump thread
5/8 – Bump thread
5/13 – Tweet again asking for a correction
5/14 – Submit site feedback via two different forms
5/14 – Tweet frustration at publication
5/15 – Publication replied to form, didn’t seem to fully understand the point
5/19 – Sent a DM to the author of the article pointing to original Tweet
5/21 – Author replied saying they will fix it. Article amended to fix and clarify the error.

Twenty one days to fix is rough. Publications and journalists; please understand that a CVE ID is important to get right. If you have any questions about CVE, how it works, or the importance, please feel free to reach out. I am happy to take the time to help you.

Microsoft, CVE, MITRE, ETERNALBLUE, Headache…

2019-02-14 Update: Thanks to Chris Mills @ MSRC (@TheChrisAM), who has been working behind the scenes since this blog was published, he has brought clarity to these assignments! MSRC is still potentially touching up some additional documentation to make it easier to see these associations, but here is the definitive answer from him:

CVE-2017-0143 ShadowBrokers : EternalSynergy (Blog)
CVE-2017-0145 ShadowBrokers : EternalRomance (Blog)
CVE-2017-0144 ShadowBrokers : EternalBlue (Blog)
CVE-2017-0146 ShadowBrokers : EternalChampion (Blog)

Note that only the EternalChampion blog does not reference the associated CVE, but he is working on getting that updated. I have also recommended that MSRC update MS17-010 to use the codenames in that advisory as well. Apparently editing the actual bulletins takes a bit more work, but he’s on it! I can’t thank Chris enough for running with this and helping bring clarity to these assignments.


There was initially a lot of confusion over the Equation Group disclosure. Which were legitimate vulnerabilities, which were new, which were known, which were patched, and ultimately how they would be referred to other than their leaked nicknames. That is the purpose of The Common Vulnerabilities and Exposures project (originally Common Vulnerability Enumeration), to give a unique ID to a specific issue so that you can reference a vulnerability without question. A year and a half later? We’re still wondering apparently.

I contacted Microsoft Security Response Center (MSRC) on August 6, 2017 asking for clarification on the CVE assignment for one of the Equation Group vulnerabilities codenamed ETERNALBLUE, because their own resources contradicted each other. From my mail:

Per an older blog [1], the vulnerability known as ‘EternalBlue’ is assigned CVE-2017-0145. From the blog:

However, in this unique case, the ransomware perpetrators used
publicly available exploit code for the patched SMB “EternalBlue”
vulnerability, CVE-2017-0145, which can be triggered by sending a
specially crafted packet to a targeted SMBv1 server.

A newer blog [2] now lists it as CVE-2017-0144, which I believe to be incorrect. From the blog:

The new ransomware can also spread using an exploit for the Server
Message Block (SMB) vulnerability CVE-2017-0144 (also known as
EternalBlue), which was fixed in security update MS17-010 and was
also exploited by WannaCrypt to spread to out-of-date machines.

Can you confirm the correct assignment for ‘EternanBlue’ [sic], and due to the second blog, the assignment for ‘EternalRomance’, and update your blog(s) accordingly?

All this time later? MSRC never answered my mail, and never fixed one of the two blogs. CVE’s description of each does not mention the nickname in either entry. So the assigning CVE Numbering Authority (Microsoft), or CNA, and the core CVE project (MITRE) still don’t answer this question. To date, the Microsoft advisories for those two CVE ID still don’t mention the nickname. To add more confusion? Try using Google to find it, and you get a third CVE ID it may be (screenshot below). Although, that one result doesn’t actually have ‘EternalBlue’ in it, making us wonder why it is the sole result. The blog that MSRC originally published to add some clarity to the Equation Group still only references MS17-010 (and a dead link now). Looking at the new location for MS17-010 doesn’t find the nickname in the advisory either.

To this day, I am still fairly sure ETERNABLUE is CVE-2017-0145 and attribute it as such, but it sure would be nice if MSRC would clean up and clarify this mess.

Further, I have had to chase down two more errant CVE assignments by MSRC in the last months, which was fairly painful. After getting the runaround on both, being told to go ask Microsoft Support via a forum (despite MSRC being the definitive source for this information), not getting a reply, opening a new ticket with MSRC, reminding them that I was still waiting… those two finally got resolved after a month or more. I really don’t like casting shade on MSRC as over the years, in total, they have been wonderful to deal with. However, the last couple of years have seen a serious decline in this type of incident which should be ‘Vulnerability 101’, and a serious uptick in their resistance to clarify assignments when asked. Finally, if you are wondering why MITRE doesn’t provide some kind of oversight to this? Well they basically never have despite repeated requests for just that. Their only oversight is a ‘CNA Report Card’ that is more about statistics of assignments and such, and does not deal with the quality of assignments, incidents of confusion like this, or anything else that would be helpful to the community.

The only upside to all of this? I got to [sic] my own typo from the quoted email.

A Samsung Galaxy 8, Phantom Notifications, and @Tmobile’s Dreadful Support

This is a blog of two topics. The first, a brief technical explanation of a problem with my Samsung phone after an upgrade to Android 8.0 (Oreo) pushed by T-Mobile, the subsequent debugging, and hopefully help for anyone else experiencing the issue. The second, my horrible experience with T-Mobile Twitter-based tech support.


On April 2, T-Mobile pushed an over-the-air update for my Samsung Galaxy 8 (G8) phone. In addition to a routine Android security patch level update, it also upgraded the phone to Android version 8, code-named Oreo. Shortly after the update, I started getting what I called ‘phantom notifications’, between one and six of them every hour or less. These were audible notifications that didn’t correspond with any discernible event on the phone, sometimes in quick succession. Over the course of a week, there were a few times where an icon would appear in the notification bar for a split second, making me think it was related to a specific event, but I couldn’t figure out what. I engaged with T-Mobile on Twitter, and they offered some ideas. Here is everything I did to debug and figure this out, based on their questions and my own ideas.

  • T-Mobile: SMS App Clear Data/Cache (I suspected it may be related to SMS)
  • Me: Full power cycle
  • Me: Changed default notification to determine if the phantoms are using system notification preferences (they are)
  • T-Mobile: Verify Notification Reminder functionality = OFF
  • T-Mobile: Verify no wireless/bluetooth/NFC turned on during phantoms
  • T-Mobile: Clear cache partition on phone via Debug menu
  • Verified software versions for all functionality (‘About Device’)
  • T-Mobile: Verify all apps are updated via play store
  • T-Mobile: Verify no apps from unknown sources
  • T-Mobile: Enable Developer options (did not change anything)
  • T-Mobile: Device Maintenance showed no app crashes, no hint of a problem
  • T-Mobile / Me: Phantom notifications do NOT vibrate, while SMS is configured to (so not SMS)
  • T-Mobile: No SD card in phone
  • T-Mobile: Uninstall Samsung Health (they suspected app causing this, that app isn’t on the phone)
  • T-Mobile: Backup SMS and clear all of the messages
  • Me: DND mode suppresses the phantom notifications (observation)
  • T-Mobile: Confirm I did not download ANY new apps on Sunday (day before update), Monday (day of update), or Tue – Thur (after update)
  • T-Mobile: Confirm the last time my phone worked w/o phantom notifications was Sunday and Monday before the patch (and every day prior since buying the phone)
  • Me: twice out of hundreds of times, i have seen a ‘health monitor’ type icon appear in notifications for a split second when it happens
  • Me: One-by-one disable app notifications, wait for phantom. process of elimination = found the offending app = PROBLEM SOLVED

Naturally, it was the last app on the list I had notifications enabled for. “Weather & Clock Widget for Android” by Devexpert.NET, which worked fine on Android 7.x, started causing these phantom notifications on Android 8.0. Uninstalling and re-installing did not fix it. The only reason I had allowed notifications from this app, is it would put the current temperature in the notification bar at all times. Blocking notifications for this app didn’t allow this behavior, but also stopped the phantom notifications. No factory reset needed.


Part 2; My dreadful experience with @Tmobile tech support via Twitter DM.

First, this isn’t the first time I have Tweeted and had them reach out via DM, offering support. I don’t recall having a good experience with them before, and this time certainly takes the cake on a poor experience. I am writing this up as a warning to others who might go this route, and as feedback to T-Mobile so they better understand what it is like on the customer side, and offer some tips for improving.

Perhaps the biggest problem with T-Mobile Twitter support, is their system for interacting with customers appears to be designed to resolve issues very quickly. I can’t speak to their workload, average customer engagement time, etc. But for a case like mine? I went through 22 different people over the course of seven days. On April 8, there were nine different people that cycled through to ‘help’ me. On April 7, while working with Reggie (who happened to be the only one out of 21 that I felt was truly helpful), he said he needed to AFK for 15 minutes for break, implying that someone else would take over. By that point, I knew I had already gone through seven others, so I told him I would happily wait until he returned. This high turnover rate on support staff worked against the process entirely for my case. Each time, the new person had to try to read the thread and figure out what was going on, and they rarely skimmed the thread it seemed. When I was offered a summary of my problem by the new person, it was typically wrong or left out important bits. T-Mobile needs to better identify problems that can’t be solved in ten minutes, and keep one or a few people on the case for consistency. When a customer repeatedly asks for a specific support person to re-engage, listen to them. Here is the list of people I dealt with:

  • Apr 3 – Joel Bannister
  • Apr 3 – Harley Sumida
  • Apr 3 – Ruben Hernandez
  • Apr 3 – Dee Medina
  • Apr 3 – Zach Ricketts
  • Apr 3 – Kimmi Smith
  • Apr 3 – Victor Loya
  • Apr 7 – Reggie Reese
  • Apr 7 – Harley Sumida
  • Apr 8 – Lauren Chan
  • Apr 8 – Pete Harman
  • Apr 8 – Marva Biggar
  • Apr 8 – Sora Yi
  • Apr 8 – Marva Biggar
  • Apr 8 – Kate Tomallo
  • Apr 8 – Lauren Chan
  • Apr 8 – Meghan Parks
  • Apr 8 – Eddie Gough
  • Apr 8 – Scott Degelman
  • Apr 8 – Ray Butler
  • Apr 9 – Dee Medina
  • Apr 9 – Mike Perez
  • Apr 9 – Alex Kimbrell
  • Apr 9 – Zach Ricketts
  • Apr 10 – PoxMaphixat [1]
  • Apr 10 – Kyle Saragosa
  • Apr 10 – Scott Degelman

[1] This was the only person that didn’t appear in Twitter DMs with a real name shown by Twitter:


The next bigger problem I faced, is that T-Mobile’s documentation for their support staff is out of date. It’s as if they had never debugged an issue on a Galaxy 8, despite them selling it for half a year. During the ordeal of figuring out my problem, I ran into several times where support failed related to this:

  • Apr 3 – Document for changing SMS message sounds is outdated, not correct for G8 (you apparently can’t on this model)
  • Apr 3 – T-Mobile said to set up a notification log for debugging purposes, yet G8 removed that functionality (ridiculous)
  • Apr 7 – The location of the ‘build number’ to enter developer mode is different on the G8 than previous models
  • Apr 7 – They asked me to go to the ‘Security’ screen in options, yet on the G8 that is ‘Lock Screen and Security’
  • Apr 7 – T-Mobile diagnostic data said ‘apps from unknown sources’ was enabled, my screen said it was disabled
  • Apr 8 – They asked me to check the ‘Samsung Health’ app (there is none, apparently part of the ‘Activity Zone’ app, but that function is disabled)
  • Apr 9 – T-Mobile kept telling me a factory reset is the way to fix this, despite it not necessarily working
  • Apr 10 – T-Mobile told me a factory reset is the way to go AFTER I solved the problem (WTF?!)

After having to correct the T-Mobile support staff this many times, and figure out how to find what they were looking for, it shows an obvious gap in their support ability. As someone who wrote my fair share of technical documentation, I cannot stress how important this is.

As mentioned above, when a new support person steps in, they have to skim the thread to catch up. One person told me that they take extensive notes to alleviate that problem, but after most of the new people offering me a summary got major parts wrong, I don’t think that is the case. Even if they do take notes, I think they are not consolidated, not done in a way for easy transition of the case, and generally convoluted. This causes the support staff to repeat the same things, ask the same questions, and waste customer time.

Next, T-Mobile needs to make sure their employees understand policy. Compare:

  • Apr 3 (Vinny) – “Thanks a bunch for remaining engaged with us at T-Force today, my name is Vinny and I’ll be taking over from here, as Krystn, as she had to step away.”
  • Apr 3 (Joel) – “Thank you so much for reaching out to T-Force! My name is Joel and I will be your #MagentaExpert!”
  • Apr 3 (Ruben) – “I hope you are having an amazing day. My name is Ruben and I will be taking excellent care of you and all of your concerns/questions today.”
  • Apr 3 (Zach) – “Thanks for sticking with us here. My name is Zach, and I’ll be taking over from here.”
  • Apr 7 (Reggie) – “I do want to introduce myself, my name is Reggie and I will be your #MagentaExpert today.”
  • Apr 8 (Meghan) – “My teammate had to step out for a quick meeting but my name is Meghan and I’ll be taking over to provide you with excellent service!”
  • Apr 8 (Eddie) – “Fun fact, Since T-Force is a team and constantly changes to ensure that customers always have support 24/7 we are not supposed to share our name since it already shows on the message.”

After support staff introduced themselves by name six times, Eddie came along and said they aren’t supposed to share their name. He further points out that Twitter shows their name (in the native web interface, not in Tweetdeck BTW), and yet that isn’t the case either as seen by “PoxMaphixat” above.

While some that interact with T-Mobile may say they are really ‘nice’, to me, that isn’t the case. Their overboard attempts to portray a fun and friendly atmosphere are insulting and a waste of time. Throughout the week, I was assured that they were there to help and resolve my issue, while not reading the prior messages, not understanding the issue, and bouncing in and out of my ticket to the point it was difficult keeping up with them. The phrase they loved to over-use, “I will be your #MagentaExpert!” is a joke. Seven days to figure out my problem, and they never did, I had to. Other phrases they love to say, adding fluff and not actual support, while not reading the thread and repeating the same things over and over:

  • I absolutely want to be able to help you in any way that I can!
  • It’s great seeing you here today. I hope you are having an amazing day.
  • That is an awesome question and definitely not something I am familiar with, but we can definitely work together to look into it!
  • I honestly want the best and fastest resolution for you!
  • Thank you for taking time out of your day on this!
  • Here at T-Force, we value customers time and always want to get them the best resolution possible without wasting their time.
  • We’ve got your back! (T-Mobile needs to remove this from their playbook, it is insulting.)
  • I really appreciate you reaching out and working with T-Force today.

Overall, I need a lot less of this fluffy wording, and a lot more I didn’t quote, and more actual support. If you have to keep telling me you “have my back” and want to give me the “best resolution possible”, you are convincing me you aren’t good at your job. We expect customer support to do that already.

Apr 3 (Joel) – “If you prefer to not do that, then you always have the option to back up the device and reset the software completely.”
Apr 3 (Zach) – “Can you please tell me if you’ve completed a master reset on the device since the update?”
Apr 3 (me) – “If a ‘master rest’ means a ‘factory reset’, that may be a deal breaker.”
Apr 3 (Zach) – “Typically, if there are any bugs that come across after an update, which this one may just be, a factory reset would be the best possible solution, as inconvenient as it can be to set everything up again.”
Apr 3 (Kimmi) – “In those instances the only fix I’ve been able to locate based on user feedback is a factory reset of the device.”
Apr 3 (Kimmi) – “Unfortunately the only option we have at this time is to complete the reset.”
Apr 3 (Victor) – “The master reset would be a great way to fix the issue in case it’s just some sort of temporary issue. ”
Apr 7 (Reggie) – “By no means do I want to tell you that you absolutely must do this, but in the end I want to respect your time and I feel like at this point the Master reset might fix the issue permanently whereas what we have done has demonstrably had no effect on the issue at hand.”
Apr 7 (me) – “If a factory reset is the answer, then I walk from Tmobile and go on a social media campaign to dissuade people from using Tmobile, because that is just sloppy programming and a complete breakdown of tech / customer support.”
Apr 8 (Marva) – “I know Reggie mentioned a master reset and that seems to be the only thing we haven’t tried up until this point, is that correct?”
Apr 8 (me) – “Safe mode has not been tried, and a reset, the nuclear option, is out of the question.”
Apr 8 (Sora) – ” I know that you do not want to do a master reset … I totally follow your logic; I do want to mention that if the software update is giving this error, then a master reset does allow the software to be restored on your phone properly.”
Apr 8 (Marva) – “The next step in troubleshooting is to complete that master reset.”
Apr 8 (Kate) – “The Master Reset sounds nuclear, but truly is the faster and cleanest resolution available.”
Apr 8 (me) – “As I said earlier this week, a factory reset means I will no longer be a T-Mobile customer, and will blog about this entire mess, that T-Mobile sent faulty software and could not debug it, and now is pressuring me to go that route while ignoring my direct questions about Samsung Health buginess, that icon that shows sometimes, and my desire to explore that route. That said, do you still think a factory reset is the right option instead of pursuing valid leads that may fix this without a reset?”
Apr 8 (me) – “From there, process of elimination can tell likely tell us which app is causing them. No safe mode, no factory reset. Please add this to your CS playbook.”
Apr 8 (Eddie) – “With the awesome software that we have nowadays, a master reset is the best option since there’s a high chance the bug will be deleted, and your information will be downloaded onto your phone within less than one hour if it’s backed up”
Apr 8 (me) – “Ugh, STOP. Do not recommend a factory reset to me again. I just gave a viable option to better figure this out that will take a few hours, and you go back to factory reset, after I have REPEATEDLY said that is a nuclear option and I a) will not do it OR b) do it and no longer be a tmobile customer.”
Apr 8 (Eddie) – “I just wanted to assure you that we are going to be here for you until we get a resolution. Never wanted to tell you that you should do a master reset.”
Apr 8 (me) – “I mentioned I found a new solution to this kind of problem, to add to your play book. And you immediately recommend a factory reset despite me REPEATEDLY saying ‘no’. You understand no means no right? I am tired of being told why a master reset is the option, and I am *more* tired of Tmobile reps not reading why it is NOT necessarily the right option, why it is NOT a guarantee it will fix anything.”
Apr 9 (Alex) – “If so, have you installed them and reinstalled them? Those are the first two steps, so let me know how that goes!”
Apr 9 (me) – “Two? There were *19* people on the Tmobile side during the course of this investigation, all of who gave up and told me to factory reset.”
Apr 9 (Alex) – “Now, I know we mentioned a master reset was something we should try.”
Apr 9 (me) – “Pretty much confirmed, “Weather & Clock Widget for Android” by http://Devexpert.NET is the one causing the phantom notifications. Uninstalling and re-installing it to start.”
Apr 9 (me) – “Uninstall & Reinstall did not fix it. So there is some weird issue between the app and the Oreo update. I can get around this by disabling notifications for that app, which only makes it so I don’t get the temperature in my notification bar. With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “I also explicitly said last night to STOP telling me to factory reset.”
Apr 9 (me) – “I have asked half a dozen times and every single one of you jerks ignore me. Focus on THAT problem instead of a factory reset.”
Apr 9 (me) – “With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “At this point i am 99.99% sure I have this resolved, again, without a factory reset.”
Apr 10 (PoxMaphixat) – “Resetting the device and processing a warranty exchange is our last resort. Which would result in a device that is fully reset as well. This might be the thing we would need to do since we’re not able to resolve this phantom issue.”
Apr 10 (me) – “Not only have i solved the issue, I have said repeatedly NOT to recommend a factory reset to me, and you assholes keep doing it. NO MEANS NO.”
Apr 10 (Kyle) – “We can see that you’ve invested a lot of time with these issues on your phone and wanted to avoid going through the previous steps that’s you’ve already done, which is why we were looking at the master reset as a last resort … So our troubleshooting steps would basically be the master reset as well though I Samsung may have more support on what’s going on with this app.”
Apr 10 (me) – “Seriously? You suggest a master reset AGAIN when I have said over and over NOT to tell me that? I solved the phantom notification issue without a reset,”
Apr 10 (Kyle) – “I would reach out to Samsung as I completely understand your concern regarding the reset and they would be able to support the app even further. Does this make sense, Brian?”
Apr 10 (me) – “You said ‘reset’ again. How can I be any more clear here? Never, EVER, not a single time, EVER tell me to factory reset my device. Don’t even mention the word ‘reset’, let alone ‘master reset’ or ‘factory reset’. I honestly feel like there is a den of rapists and molesters working at Tmobile, who don’t understand what the word ‘NO’ means. Does this make sense, Joel / Harley / Ruben / Dee / Zach / Kimmi / Victor / Lauren / Pete / Marva / Kate / Meghan / Eddie / Scott / Ray / Mike / Alex / Zach / PoxMaphixat / Kyle?”

After this? Scott said ‘reset’ once more shortly after my last message. This is the text-book definition of the worst customer support that can be offered. A customer specifically says, over and over, not to recommend a bad support option (the factory reset). Yet, T-Mobile kept recommending it every single time. It gets to the point where it is a trigger word for me, because it clearly shows the support person didn’t read the prior messages. It means that the support staff didn’t leave a message for the next person not to bring up a factory reset. Worse? I SOLVED the technical issue, without a factory reset, and said as much. T-Mobile’s solution? Keep recommending a factory reset anyway, when it was clearly not needed. This is hands-down the worst customer service you could possibly offer, and completely insensitive to a customer. I don’t really care where the breakdown happened, other than it happened half a dozen times, but when a customer says “do not do $thing“, you should NOT do $thing. No questions, no arguments, no equivocation. Yet T-Mobile ignored that basic point, that basic understanding of the tenets of customer support. 18 separate times, reset was their answer, three times after resolving my issue.

My next advice for T-Mobile is to embrace an old classic of customer service. Over six days, interacting with 21 different support people, after repeated complaints about many of them, no manager stepped in. At least, no one identified themselves as a manager, no one exhibited any signs they were a manager, and absolutely no one made it a point to get me a resolution other than the empty “we’ve got your back” lies. Imagine going into a Taco Bell and talking with 21 employees trying to resolve a problem, that your Mexican Pizza was missing ingredients or not cooked, and that entire time no manager stepping in to ensure you got a properly prepared and cooked food item. To me, the customer, those scenarios are no different.

Finally, the bigger picture. I engaged support for one problem, the phantom notifications, which I eventually resolved myself. During the process, T-Mobile asked me questions that highlighted other problems. Despite figuring out the original, I left the engagement with two additional problems that they did not resolve. First, I asked how to disabled ‘Bixby’ completely, and they couldn’t help. Like so many other things, they didn’t understand the software, and/or their documentation wasn’t updated. I had to tell them to disable it per their instructions, it required creating a Samsung account. You actually can’t access the real settings of that malware without creating an account. That is atrocious and just bad design. Second, when we went down the road of the occasional phantom notification icon that I saw, it led us to the ‘Samsung Health’ feature within ‘Activity Zone’. On my phone, it says “tap here to get started” and tapping there does nothing. T-Mobile never helped with that, and after specifically asking them to half a dozen times, they told me to talk to Samsung.

Two more bonus observations, that came up during this ordeal. First, the T-Mobile software update downloaded over 4G, not WiFi. It used to prompt you if you wanted to wait for WiFi and this time it did not. Second, I mentioned that T-Mobile was still sending SMS notifications to me before 9 AM, and one of the support people were gung-ho saying that was not right, they would take my complaint to the top! Well, good luck there, since the last time I brought that issue up on Twitter it did go to the top, all the way to the office of the executives. Nothing ever came of it and I still get text messages from them before 9 AM. If you are going to grab that flag and head on a crusade on my behalf? Maybe consider better helping fix my original problem first.

So, T-Mobile, I have given you a wide variety of ideas for improving customer support. It is in the context of a support case you can easily reference. These ideas are very much in line with many other support services offered by similar services and companies. It’s time for you to up your game.

It’s 2016, why is rotating a video such a pain?

How many times have you quickly shot a video on your phone and not rotated it for landscape? It happens too often and we see these videos all over social media. I sometimes forget to do it as well, or portrait is more in line with what I am shooting. So, I want to quickly rotate a video 90 degrees sometimes. Should be easy, right?

I’ve asked friends and social media before, but I asked again last night and got a lot of great input. My criteria were very simple, but I did not specify platform; I want to load an MP4 video, rotate it 90 degrees, and save it. I didn’t qualify it, but my expectations are that it would not lose quality, it would keep the original MP4 format, and that the process was “one-click” (or close). While I have plenty of history using Linux, going back to CLI graphics tools to do this is not ideal for me, but I considered those options.

  • @cl suggested Windows Movie Maker – It will rotate trivially, but saves your MP4 as WMV and the quality drops noticeably.
  • @TCMBC suggested mencoder – A command line utility, part of MPlayer. So it is not trivial (download, configure, compile, figure out CLI syntax), but it does rotate. Yet, the quality drops noticeably.
  • @viss suggested ffmpeg – A command line utility and graphics library, not so trivial. It did rotate, but the quality drops noticeably.
  • @viss suggested The ‘Rotate My Video‘ web site – It is a bit slow for file upload and conversion, but very easy to use. It played the video correctly in my browser, but when I saved the video the final copy was not rotated.
  • @DeviantOllam suggested (in DM) the Rotate Video FX app for Android – I thought the UX wasn’t intuitive for starters. It did rotate the video for immediate playback, but no apparently way to save the new video back to the device. Sharing it brings up the usual Android options, but uploading the video to google drive and the video was not rotated.
  • @elkentaro suggested Apple’s QuickTime Player – Even with his reference which is outdated, there is no apparent rotation function. Even the ability to save a file is now ‘Pro’ only.
  • MegaManSec suggested ImageMagick ‘convert’ utility – this didn’t work and gave me a nice reminder of the old ‘terminal flash attacks’ from the early 90s.
  • @DeviantOllum suggested Virtual Dub but warned me that some versions handle MP4 and some don’t. Thus, I didn’t try it.
  • @Grifter801 suggested VLC but qualified it “just for viewing”.
  • @mehebner suggested Open Shot Video Player but said it is Linux only, which isn’t convenient.
  • @cl suggested iMovie but it is Mac OS X only, which isn’t convenient.
  • @cl suggested Facebook but he isn’t sure you can save after. I am fairly sure you lose quality though.

The final recommendation, and the one that worked the best for me, is Handbrake suggested by @bmirvine. The upside is I had it installed (but an old version) and am familiar with it to a degree. The best part about conversion is that the video does not lose any quality. The downside is trying to figure out the ‘Extra Option’ argument to rotate is a raging mess, as seen on this thread. I found that using “, –rotate=4” as the extra option worked for version 0.10.5.0 64-bit (latest as of this blog). The only other annoyance is that Windows won’t show a thumbnail of the newly saved video for some reason. [Update: with a newer version of the K-Lite codec pack, the thumbnails render fine.]

There are my quick testing results. I hope it helps. I’d like to give a big round of thanks to all who contributed ideas late night. Reminds me that Twitter has some value and isn’t a cesspool of insipid political tripe. =)