CVE and the matter of “unique” ID numbers

Common Vulnerability Enumeration, now known as Common Vulnerabilities and Exposures (CVE) is a vulnerability database (ignore their silly claim to be a ‘dictionary’) that the information security industry relies on heavily, unfortunately. Per MITRE’s CVE page, “CVE® is a list of entries—each containing an identification number, a description, and at least one public reference—for publicly known cybersecurity vulnerabilities.” Great, digging a bit deeper into their ‘About‘ page, we get a better definition:

Use of CVE Entries, which are assigned by CVE Numbering Authorities (CNAs) from around the world, ensures confidence among parties when used to discuss or share information about a unique software or firmware vulnerability, provides a baseline for tool evaluation, and enables data exchange for cybersecurity automation.

Please take note that a CVE Entry, or ID number, “ensures confidence” when discussing or sharing information about a vulnerability. Basically, it is supposed to be a unique ID to ensure that confidence. Despite that, any of my dozen loyal Twitter followers will see me constantly pinging researchers, vendors, and the media pointing out that they are using the wrong CVE number to reference a vulnerability. Often times it is a case of not copying and pasting, rather typing it out manually. It is also why in the vulnerability database (VDB) world, we strongly emphasize that copy/paste is the best thing to do in order to prevent transcription errors on an ID that is supposed to be unique.

Sure, it seems pedantic to a degree, but imagine if your doctor decided to manually transcribe a diagnosis code after your visit and you get a call saying you were diagnosed with something completely different. In the vulnerability world, it means you might be vulnerable to something and have no idea if so. If you are, you aren’t sure if there is a solution. Maybe a bit of a dramatic analogy? But.. it holds water, has a bit of recent history, and is the kind that helps administrators better understand the underlying issue.

Instead of sending out a series of Tweets on the latest example, I decided to write a blog to show how these little typos can snowball quickly. Any mature VDB will have a variety of processes to catch wind of a CVE ID that they haven’t seen before. It can be as simple as a live search on Twitter for ‘CVE’ (super noisy) or more technical means. If you run across an unknown CVE you Google it to start, that simple. Today’s example was CVE-2019-0895, which appeared to be a “new windows zero-day”. Exciting in the world of VDBs!

Let me go ahead and spoil things, to make this easier. These articles call it “2019-0895”, but in reality, they actually mean “2019-0859”. A simple transposition of numbers, which is all too common in prior cases. Based on ten-second review, it appears that Fossbytes was the first to transcribe these numbers (Unverified @fossbytes14 on Twitter?). A day later, extremely similar articles appeared on Prodefense (no Twitter and broken Contact form?) and “In Depth IT News / SecNews” which has some serious rendering issues in Chrome. The day after that, Tech Rights references it via a weird embedded link below in an unrelated article [1], and Tux Machines posted about it with key quotes cribbed from other articles, the Fossbytes article in this case.

In each case, it is clear that the offending typo came from Fossbytes. The “In Depth IT News” site even links to https://securelist.com/new-win32k-zero-day-cve-2019-0859/90435/ which has the correct CVE ID in the URL. It is absolutely clear that most of these sites are using automated crap to aggregate content and have no real desire to share accurate news. Each one of them are evidence to the damage caused by a single transposition error from Fossbytes, a “leading source of technology news with a focus on Linux distro releases” … that decided it was important to write about this critical Windows zero day? A critical zero day that is actually ten days old at the time of their article.

OK, hopefully we’re all on the same page here. My Twitter feed is a small graveyard of similar examples from the past few years. Each and every time, the “news” organizations that spread these bad IDs and introduce confusion and questions into the equation, and are the antithesis of a “news” site. Finally, I would like to go on the record about one more bit regarding CVE, which will come as no surprise. On the CVE ‘About’ page, it says CVE is:

Industry-endorsed via the CVE Numbering Authorities, CVE Board, and numerous products and services that include CVE

As a former ten-year veteran of the CVE Board, I do not endorse CVE.


[0] Note: If any of my links show a fixed version of the CVE, good! You can see the originals on archive.today.
[1] This should really be a separate blog post, but it would mostly be cursing around a simple concept; this is the problem with content/link aggregation sites… which are a plague on the Internet. In 2019, they aren’t trying to help, they are desperate attempts to make a few bucks. Disagree? They would have caught this error when they did a quick tech edit pass on the article. But they didn’t, because it is all automated and centered around ‘SEO’ (search engine optimization) so it appears in Google results and you click and see the ads they are serving. I bet if anyone dug deep on such sites, the amount of questionable traffic or malware they delivered might be enlightening. Go back to where this is linked from and notice the URL of the article (/2019/04/18/libreoffice-6-2-3/) and how far you have to scroll to get to the bottom of the page, past all the “content”.

Microsoft, CVE, MITRE, ETERNALBLUE, Headache…

2019-02-14 Update: Thanks to Chris Mills @ MSRC (@TheChrisAM), who has been working behind the scenes since this blog was published, he has brought clarity to these assignments! MSRC is still potentially touching up some additional documentation to make it easier to see these associations, but here is the definitive answer from him:

CVE-2017-0143 ShadowBrokers : EternalSynergy (Blog)
CVE-2017-0145 ShadowBrokers : EternalRomance (Blog)
CVE-2017-0144 ShadowBrokers : EternalBlue (Blog)
CVE-2017-0146 ShadowBrokers : EternalChampion (Blog)

Note that only the EternalChampion blog does not reference the associated CVE, but he is working on getting that updated. I have also recommended that MSRC update MS17-010 to use the codenames in that advisory as well. Apparently editing the actual bulletins takes a bit more work, but he’s on it! I can’t thank Chris enough for running with this and helping bring clarity to these assignments.


There was initially a lot of confusion over the Equation Group disclosure. Which were legitimate vulnerabilities, which were new, which were known, which were patched, and ultimately how they would be referred to other than their leaked nicknames. That is the purpose of The Common Vulnerabilities and Exposures project (originally Common Vulnerability Enumeration), to give a unique ID to a specific issue so that you can reference a vulnerability without question. A year and a half later? We’re still wondering apparently.

I contacted Microsoft Security Response Center (MSRC) on August 6, 2017 asking for clarification on the CVE assignment for one of the Equation Group vulnerabilities codenamed ETERNALBLUE, because their own resources contradicted each other. From my mail:

Per an older blog [1], the vulnerability known as ‘EternalBlue’ is assigned CVE-2017-0145. From the blog:

However, in this unique case, the ransomware perpetrators used
publicly available exploit code for the patched SMB “EternalBlue”
vulnerability, CVE-2017-0145, which can be triggered by sending a
specially crafted packet to a targeted SMBv1 server.

A newer blog [2] now lists it as CVE-2017-0144, which I believe to be incorrect. From the blog:

The new ransomware can also spread using an exploit for the Server
Message Block (SMB) vulnerability CVE-2017-0144 (also known as
EternalBlue), which was fixed in security update MS17-010 and was
also exploited by WannaCrypt to spread to out-of-date machines.

Can you confirm the correct assignment for ‘EternanBlue’ [sic], and due to the second blog, the assignment for ‘EternalRomance’, and update your blog(s) accordingly?

All this time later? MSRC never answered my mail, and never fixed one of the two blogs. CVE’s description of each does not mention the nickname in either entry. So the assigning CVE Numbering Authority (Microsoft), or CNA, and the core CVE project (MITRE) still don’t answer this question. To date, the Microsoft advisories for those two CVE ID still don’t mention the nickname. To add more confusion? Try using Google to find it, and you get a third CVE ID it may be (screenshot below). Although, that one result doesn’t actually have ‘EternalBlue’ in it, making us wonder why it is the sole result. The blog that MSRC originally published to add some clarity to the Equation Group still only references MS17-010 (and a dead link now). Looking at the new location for MS17-010 doesn’t find the nickname in the advisory either.

To this day, I am still fairly sure ETERNABLUE is CVE-2017-0145 and attribute it as such, but it sure would be nice if MSRC would clean up and clarify this mess.

Further, I have had to chase down two more errant CVE assignments by MSRC in the last months, which was fairly painful. After getting the runaround on both, being told to go ask Microsoft Support via a forum (despite MSRC being the definitive source for this information), not getting a reply, opening a new ticket with MSRC, reminding them that I was still waiting… those two finally got resolved after a month or more. I really don’t like casting shade on MSRC as over the years, in total, they have been wonderful to deal with. However, the last couple of years have seen a serious decline in this type of incident which should be ‘Vulnerability 101’, and a serious uptick in their resistance to clarify assignments when asked. Finally, if you are wondering why MITRE doesn’t provide some kind of oversight to this? Well they basically never have despite repeated requests for just that. Their only oversight is a ‘CNA Report Card’ that is more about statistics of assignments and such, and does not deal with the quality of assignments, incidents of confusion like this, or anything else that would be helpful to the community.

The only upside to all of this? I got to [sic] my own typo from the quoted email.

The Attrition DC26 Badge Challenge Post Mortem

This year, which was my final trip to DEF CON, I made up one last round of Attrition DEF CON badges. In prior years they were typically engraved luggage tags a bit more specific to the year:

Since #BadgeLife has become a big thing, especially this year as far as I can tell, I decided to go a bit lower rent on the badge material but ‘up the game’ on the content. I did a ‘cipher challenge’, which of course was never meant to be a real challenge. I’m not nearly smart enough for that shit. I literally came up with it in less than a day, didn’t vet it with anyone, and just moved to mock up a badge and print. Because I am so pro! I also figured anyone who knows me would know not to trust me on anything ‘cipher’ or ‘challenge’, especially ‘cipher challenge’. Unfortunately, and I do feel bad, a handful of badge-holders went down this rabbit hole.

This write-up is for them, to explain just how fast this was put together, and the lessons I learned as well. The cliff notes details, as I originally intended:

  1. https://en.wikipedia.org/wiki/Cirth (hobbit) -> “never trust us”
  2. https://en.wikipedia.org/wiki/Wingdings -> “except this time”
  3. location hint (flamingo hotel) -> “Phoenicopteriformes”
  4. refined location – wildlife habitat long/lat -> 36.11662720392657 / -115.17115294683322
  5. 08/11/2018 @ 3:04am (UTC) Epoch Unix Time -> “1533956647”
  6. Klingon “take proof you were there” -> “pa’ SoH’a’ tob tlhap”
  7. random letters/numbers -> (unsolvable/gibberish)
  8. show Jericho proof (latin) -> ostende inamabilis sciurus
  9. winner winner chicken dinner -> (icons)

Seems pretty straight forward! Unfortunately, a few of these didn’t work out so well as I found out, in surprising ways. Here are the hiccups I didn’t expect.

  • (1) There are multiple Cirth character sets. Pretty minor, but it led to a couple people saying the translation was off. Worse? That one character that was off fed into another hint and made it more believable. I should have read through the Wikipedia article to notice that, but growing up as a skilled writer in ‘Tolkein Runic’ (Cirth), I didn’t think about it.
  • (2) Always trust the first hint, never the second!
  • (5) So… Epoch Unix Time is an absolute. You don’t adjust for timezones, because the time is in Coordinated Universal Time (UTC). The Wikipedia entry for UTC confirms it “is not adjusted for daylight saving time“. So my intention of it being on Saturday morning at 3:04am was correct. I didn’t account for everyone adjusting for time zones. I also didn’t account for some adjusting for Las Vegas’ time zone (Pacific) or trying to second-guess it and using my time zone (Mountain). At this point I am vindicated, anyone loitering around flamingos at the Flamingo between ~ 8p – 10p local time, were not following the cipher. Yes, I still feel bad they showed up thinking there was a prize/reward there.
  • (6) I really should have known better here, since Google Translate fails to translate simple text from one language to another, and then back again. I fell to this trap using the first Klingon translator that Google offered and did a simple one-way translation. Unfortunately, that same site changed “take proof you were there” drastically to involve something with a cat in it. I like cats, everyone knows this, so the clue still had some crazy merit. Fortunately for me, one of the badge-holders knows a lot more about Klingon than the online translators do, and gave me a deserved verbal beratement over the horrible translation. This led me back to that translator, where I pasted “pa’ SoH’a’ tob tlhap” back into it and got, you guessed it… “you take a cat room“. This was a solid break in the intended chain, and a deal breaker for solving the badge. Oops.
  • (7) This line had a simple intention. This line may have been the weirdest in the long run. A bunch of random numbers and letters, with no intended meaning, to be an ultimate ‘gotcha’. So no one could say they solved it, or if they did, I could challenge them on that line. I left this up to the wonderful badge artist, Anushika, who typed in a random string while designing it. Between that and the chosen font, there was even question over one or two characters. Either way, I thought it served a purpose. One nice lady from Australia (she is nice, despite her DMs irrationally suggesting I not to call her that) spent a lot of time on this, maybe more than anyone else. At one point she messaged “Threw it through successive shifts. And the answer it gave me was successive shifts.” This was after I reminded her on previous comments, that “i’m not really bright. hashed, encrypted, encoded… i get so confused“. No false modesty or deception; math is a religion, and I don’t believe. Ergo, crypto is a foreign language to me for the most part. So that random line had some merit in the math world maybe? Put it through successive shifts, and the answer is more successive shifts. That certainly sounds like I was really brilliant in a troll cipher, when I was the farthest thing from it. She kind of spooked me when she told me that and I thought “oh shit, this line has meaning?!” Kind of disappointed that a ‘troll cipher’ isn’t a real thing with a Wikipedia entry!
  • (8) Translation woes again. As someone who took a year of Latin in high school, seriously, and knows about the headache of online translators… not sure how I got burned twice in one badge. I translated “show squirrel proof” since I knew it wouldn’t handle “jericho”, and got “ostende inamabilis sciurus“. This is where it gets really weird. Someone messaged while in Vegas that the translation was off, and I went to check again, using Google Translate again. Click that link and you will see the problem. The translation changed between making the badge, and someone translating it after receiving the badge, which was around 30 – 40 days. So now it became “inamabilis sciurus ostendit probationem“. This caused a problem because the first translation now reverses as “show squirrel” which is lacking a crucial word. The updated translation, when reversed, comes back as “squirrel proof shows“, which is a bit closer to the intent. Ugh. For fun, since we had to pick ‘Latin’ nicknames in my Latin class, I chose Sylvester. #JerichoTrivia

So there you go badge-holders and adventure-seekers! I sincerely apologize for any hardship you went through, to a degree, because that first line really is gospel when it comes to me, attrition, and anything remotely close to a challenge. Years prior, I wanted to do a luggage tag badge like those pictured above, but cut out holes in a Goonies sort of way along with instructions to stand in the middle of Las Vegas Blvd to line up three landmarks to figure out where the party was. After this badge challenge? Probably for the best I didn’t, or I bet I would have gotten a few people run over. On the upside, you got to spend time with Flamingos, largely more bearable than the average DEF CON attendee.

The Uncertain Future of Necco Wafers and the Logical Response

Recently, the Necco wafer factory abruptly shut down after the company sold it to an “unknown buyer”.

The footer to that image reads: “Necco, the oldest candy company in the country, abruptly shut down its Revere, Mass. factory on July 26, and left about 230 workers jobless. (Reuters)”

Yes, the oldest candy company in the country! This is history right here. We must preserve and honor it, do everything we can to preserve it, even if a tiny majority of Americans enjoy Necco wafers (like me)! I’m not the only one… Newsweek reports, “Fans stock up as America’s Oldest Candy Company Faces Closure”.

I caught wind of this several months ago, and as a fan of Necco wafers, I was obviously worried. So I did what any red-blooded, patriotic, Type-1 diabetic American would do… I bought some.

I bought 154 rolls of Necco wafers, including the rare Sour ones that are doubly delicious.

That is 33,850 calories of Necco wafers.

That is 8,624 carbohydrates (sugar) of Necco wafers.

And my insurance provider tried to tell me and my doctor that I didn’t need insulin as a Type-1 diabetic. CHALLENGE ACCEPTED.

Jericho in Vegas Next Week… (for real)

Hi!

Given my occasional good-natured trolling on Twitter, and since many have asked me the last few weeks, I want to set the record straight. I will be in Las Vegas next week, for real. I arrive tomorrow evening and leave the following Sunday. This is the first time at BH/DC in several years for me.

Between Monday and Wednesday I will be doing the corporate thing around Mandalay and adjacent to the Black Hat event. I am not actually attending the conference, thus ‘adjacent’. Each day already has several meetings lined up so I won’t be readily available for parts of the day. When not in a meeting, happy to meet up with anyone looking to better understand the nuances of the vulnerability intelligence landscape. On Tuesday evening I will be at the Guidepoint Party at the Aureole in Mandalay Bay for several hours. Wednesday night I hope to crash the BSidesLV pool party and enjoy the cool 94 degree temperatures Vegas has to offer at night.

Between Thursday and Sunday I will be doing the hallway thing at DEF CON primarily. On Thursday at 3:30p I will be on the DC101 panel, apparently because I am old, to dish out horror stories about our industry to those attending. On Friday and Saturday I will no doubt be around Skytalks on and off to harass and support that track. Otherwise, you can likely find me roaming around Caesar’s and Flamingo checking out villages and side events.

I have a Twitter client on my phone but it doesn’t have any alerts, so that won’t be a reliable way to reach me. I hope to check Twitter every so often but my lizard brain isn’t wired to check that really. If I do camp down at a spot in a hallway or bar I hope to remember to Tweet my location in case anyone wants to discuss wildlife rehabilitation or vulnerability databases or anything else interesting really. As for spotting me, I will be one of ~ 100 wearing the DC26 Attrition badge, and a T-shirt that has an animal on it. As many have said, I too am really bad at remembering names while fairly good at remembering faces. Worse, when I do remember trying to figure out if you prefer to go by real name or handle at what events. Please don’t be offended and please re-introduce yourself! It may take me a minute to remember our history, my brain is a tad broken these days.

Finally, this will be my last year attending DEF CON. I attended DEF CON 2 back in 1994 at the Sahara, so this will be my 25th anniversary. I see a lot of value in DEF CON and continue to volunteer reviewing talks on the CFP panel to help shape the conference and try to make the content the best possible. Next year I will stay on with CFP in a more limited role, but still offer my input for certain types of talks. That said, as many say before and after ‘hacker summer camp’, the week is emotionally and physically draining, and many of us often come back with ‘con flu’ or some other kind of crud. The last time I attended, I went a full week not seeing some friends that were in Las Vegas, because the meta-convention is just so big and spread out. I hope that doesn’t happen this year, but it is one discouraging aspect of a week in Vegas.

While DEF CON doesn’t work so well for me personally, I see a lot of potential in it especially with the huge rise of villages. More and more that I talk to say that the villages are the first part of the conference that attracts them, more so than the main lineup of talks. Villages are an evolved modern evolution of old “birds of a feather” sessions at conferences back in the day, before ‘hallway con’ was a thing even. A group of people that share a particular interest and want to focus on a given topic have the ability to do it. Even better, often times that comes with elaborate and painstakingly designed networks and challenges to test your skills and learn more. In addition to villages are the side events for runners, shooters, coffee-drinkers, and more. I encourage everyone, especially newcomers, to embrace these side events and villages. DEF CON will be what you make of it, and there is more opportunity now than ever before to make the best of it.

DC26 Attrition Badge Round-up

This is the first DEF CON I am attending after a long break. For kicks I decided to make up a run of DC26 Attrition badges like prior years and conferences. Depending on who you ask, the badge is a decoration only, or it gets you into fabulous parties and amazing events. Anyone with a badge is encouraged to embellish.

Since the July 5 announcement of the badge, I increasingly focused on using them to raise money for charity. That, in turn, prompted several people to ask for details of the badges and the money raised. This blog will hopefully answer those questions and maybe inspire others to help out when they can. If you aren’t interested in the quick story, scroll down to the inspiration section please.

First, a link-heavy summary. On July 7, I did the first charity challenge looking to raise money for the ACLU, GLBT Community Center of Colorado (The Center), and Planned Parenthood. I also started giving out a a handful of personal challenges to random people expressing interest in a badge with fun results.

On July 10, I did a second charity drive bigger than the first. I also offered one badge up as part of an art challenge for the best original art featuring Lazlo. Deathjaw17 won that with this epic piece:

In addition to the art, I did a few other trades including for this slick challenge coin as well as a few other DC26 badges. At this point some of the winners of badges started posting pics, including with chickens, with epic beasts, and with bubbly! The Lazlo badge also got a tour of Philly and a sweet visit to the CompSci building in War Games. One badge went out and lead to a fun picture and backstory of a ‘dojo squirrel’. During this process, I got an unexpected care package from Kentaro, that he sent before I sent him a badge, and @Otterannihilation received a badge and sent back an amazing gift as a thanks. Meanwhile, pictures of badges kept coming:

    Inspiration and the Opposite

By this point, after two big charity drives, and several subsequent one-off drives, it was clear to me that raising money for charities was a great option. Badges were in demand and a lot of great people were willing to throw in money to help great causes. This also led to some other great opportunities that aren’t donations to charity, but amazing ways to help out. The level of inspiration and good-will in our industry is always refreshing, one of the few things that keep some of us from losing all hope. More on that later.

The opposite of inspiration came in two forms. First, while the badges w/ lanyards cost $298.60, but the postage to mail them out to x people cost $448.12, meaning the entire effort cost $746.72. This was due to the lanyards, which meant the badge couldn’t go as an envelope; they had to go as a package. Each envelope cost $3.50 domestic, $10 Canadian, and between $13.75 and $14.25 to mail international. This resulted in one fun trip to the post office that took around 30 minutes and produced a generous receipt.

The second came in the form of being questioned and challenged about my badges repeatedly, and being accused ofstrongly [reinforcing] exclusive cliques within infosec“. After assuring someone this was not “a dark stunt satirizing infosec exclusionism and signaling“, giving information on the charity contributions at the time, and reminding everyone that “the charity-driven badges are open to *anyone*. i have sent badges last week, and will send some this week, to people I don’t know and have had little to no interaction with“, I still faced questions about if I was reinforcing the exclusive cliques in infosec. I’ll say this definitively; I am not reinforcing cliques at all. This is trivial to see if you remember the definition of a ‘clique’, and consider that I don’t know half the people getting a badge other than a brief Twitter interaction.

OK, back to the inspiration. At the suggestion of Noah, with his input, two badges were given out to people who volunteered to provide InfoSec training for free. First, Jim Manico volunteered to give one of his well-known and appreciated AppSec classes in December on his birthday, for free, with the focus of recruiting women, LGBQT, and/or PoC for the class. Additionally, Bones volunteered to give design and give an infrastructure/cloud security pentesting course. I also suckered her into slipping in a not-so-subtle requirement.

An even bigger inspiration, and one that shocked me, was the community stepping up to donate to charity for a badge. Once I saw the generosity, I ran with it and focused on using a majority of the badges to continue raising money for charities I support, and ones that the donors support. The charities that received donations in return for badges included the ACLU, Cavy Care, Center for Genocide Research and Education, Colorado Animal Rescue, Electronic Frontier Foundation, Greenwood Wildlife Rehabilitation Center, Hawaiian Humane Society, Kids in Need Foundation, Planned Parenthood, Retriever Rescue of Colorado, SaveABunny, Special Operations Warrior Foundation, Sprout Therapeutic Riding and Education Center, The Wild Animal Sanctuary, and Women in Security and Privacy (WISP). A total of 69 donations from 67 heroes between 2018-07-06 and 2018-07-28, raised a total of $8453.47. I’m still happily shocked at this outcome.

I also want to thank Heidi for chatting and educating me about Women in Security and Privacy (WISP) and their initiative to help more women get to DEF CON. Over a week of chatting, it started out as “this is my first DEF CON and it is rough financially” to her being one of the recipients of the WISP grants. Even better, one of the people that donated and won a badge said to give it to someone else. I suggested Heidi and they said that was a good choice! So on top of getting help to DEF CON, she got a badge, and I threw in some stickers to round out the fun.

Finally… are you sad you didn’t get a badge? Depressed that you didn’t get a chance to donate to charity to win one? Fortunately for you, there is one last chance! Jives reached out and we’re partnering for a big charity auction, with a couple days left! You can bid to win a DerbyCon ticket, a DC26 Attrition badge, and a custom box of shit! Bid now, bid often, win this sucker

EFF Lock Screen Graphics – FYI and a Minor Touch-up to One

For those who haven’t seen, the Electronic Frontier Foundation (EFF) has created several lock-screen / wallpaper images related to protecting your rights. I wanted to use the first one on my Galaxy S8 Active, but the image interferes with seeing the clock, date, and notification icons. So I moved the text of the image down just enough so that it fits comfortably while not obscuring any information. Screenshot below, and a link to my version of the image you can download.

DEF CON 26 CFP Basic Statistics and Observations

This is the second blog in a series about DEF CON 26 CFP. The first:

A Look Into the DEF CON CFP Review Board (we’re actually really boring people)


First, this post is not sanctioned by DEF CON in any way. I am a member of the CFP team who decided to keep some rudimentary statistics on the submissions this year, as I did last year. I did this to give the team a feel for just how many submissions we got, how many talks we accepted, and primarily to track the way we voted. This greatly assists the powers that be (the amazing Nikita) to more quickly determine which talks are well-received. Due to time constraints, I was not able to track as much metadata, so this blog will be shorter than last years.

First, a few bits of information:

  • DEF CON 26 CFP opened on January 16, 2018
  • DEF CON 26 CFP closed on May 01, 2018
  • Two talks were submitted after closing date and were considered for various reasons
  • We received 551 submissions (up from 536 last year)
  • Four of the submissions were withdrawn by the submitters by the end of CFP
  • BlackHat received around 1,000 submissions this year for comparison

A recurring theme in these blogs and our Tweets throughout the CFP process is strong encouragement to submit early. While we did get a share of submissions in January and February, you can still the huge spike we experience in April (a majority a day before CFP closed), and May (on the day it closed). The two weeks between the end of CFP and the time when acceptance/rejection letters are sent out become stressful as we’re under deadline to review talks, try to get last minute feedback when we can, and make final decisions.

Of the 551 submissions, 107 were accepted (19.4%). There were 388 unique male submitters, 39 unique female submitters, and 14 anonymous submissions (note: we only catalog based on the gender, if known, of the primary speaker). Of those 14 anonymous submissions, 3 were trivially identified because the submitter didn’t scrub their submission properly or submitted work that had been presented before and was caught with a quick Google or Bing search.

Of the 551 submissions, 173 (31.40%) said they would release a new tool. 77 (13.97%) said they would release an exploit, up from 56 (10.53%) last year. Of all the submissions, 216 (39.20%) were also submitted to Black Hat and 51 (9.26%) said that speaking at DEF CON was contingent upon Black Hat accepting their talk. Only 73 (13.25%) submissions were also submitted to BSidesLV. Of the 551 submissions, 122 of the speakers had presented before at DEF CON, and an additional 28 had presented before at a DC Village or Workshop.

Unfortunately, time did not permit me to properly track ‘red’ vs ‘blue’ vs ‘black’ submissions, nor categorize the talks. That said, 11 talks were about ‘Artificial Intelligence’ and/or ‘Machine Learning’, even if some of them didn’t quite seem to know what those terms really mean. Ten submissions were on the topic of, or heavily related to, blockchain. Eight submissions came with the ultra creative title that included “for fun and profit”, four included “all your $blah belong to us”, two submissions used “pwned” in the title, and fortunately for our sanity, none wanted to make $blah great again.


That’s it! I realized this is a bit more brief than last year, but the time requirement of reviewing all of the submissions is crazy. Finding extra time to maintain the sheet is rough, and generating even more statistics or tracking additional metadata just can’t happen sometimes. Fortunately for me, this year Highwiz stepped up and did an incredible amount of work filling in data, especially while I was lost in the mountains for a few days. 

A Look Into the DEF CON CFP Review Board (we’re actually really boring people)

Written by Highwiz with contributions and editing from Jericho

Being on the DEF CON CFP Review Board can be as exciting as {something}; as frustrating as {something}; as thought provoking as {something}; and as enriching as {something}. It’s like mad libs, I hope you’ve filled in this section with something good.

Each year, myself and somewhere between 16 and 20 other reviewers take on the responsibility of selecting the best possible talks for DEF CON.

Oh, I should also apologize in advance as you read this first entry in the CFP Blog series. I apologize because I am not known for my brevity. In the “written word” and especially when it comes to something I’m passionate about, I tend to be wordy AF. [See, like that sentence: Could have just said “Hope you enjoy”, but nope – not me…].

I do genuinely hope that someone finds these blog postings helpful and that it will allow submitters (or potential submitters) some insight into the way we work so as to better prepare their submissions in the future.

In its original form, this post was about as dry as some of the white papers we read that were included in several submissions. Speaking of, white papers help tremendously when we’re reviewing your submissions, and if you include one, you’re already ahead of the pack. Sadly however, while White Papers do indeed help your chances during the CFP, they make for really shitty blog posts.

While we’re on this wildtangent of things that are related to the CFP Board but not actually part of the CFP Process itself, let’s talk about the term “CFP”. Above, I mentioned white papers; while the term CFP originally did mean “Call For Papers”, it doesn’t anymore. Most people don’t submit papers. When you think about the term CFP, you should really think of it as Call For Presentations. I know I’m not the first person to say that and I definitely won’t be the last, but still, it bears saying.

Alright, back to the topic at hand…

This year, the DEF CON Call for Presentations (CFP) Review board was made up of 16 “General Reviewers”,  six “Special Reviewers”, and two members of the DEF CON staff.

The DC CFP process is not “blind”, meaning reviewers can see each other’s votes, and we see who submitted it unless they specifically opt to stay anonymous (and properly scrub their submission). There are merits for both open review and blind review, but we’ve found that an open review significantly helps our process as there is a lot of good discussion about each individual submission. One reviewer may spend considerable time digging into the topic, researching prior disclosures or talks along the same lines, or offer their personal in-depth knowledge which typically helps several others better understand the topic and state of research.

If you submitted a talk to DEF CON this year, then all of the General Reviewers most likely reviewed and discussed your talk. While these reviewers tend to agree on many talks there are also submissions that cause arguments and intense heated discussions. Most of the review board members have a very extensive vocabulary and seem to enjoy finding new and creative ways to use the word “fuck” in a sentence (both in the positive and negative). Though, why the topic of vocabulary is at hand, let me say this to my fellow review board members: y’all motherfuckers need to find a new word besides “pedestrian“. I’ll leave it at that.

As reviewers, every year we’re often left wondering why certain people have chosen to submit to DEF CON and whether or not they actually understand what type of conference it is. A prevailing sentiment on many submissions is “This is not a DEF CON talk”. While the content may be of significant quality, the question we often ask ourselves is “is this talk right for DEF CON?”. Sometimes the answer is that while it would be good at a developer conference, RSA, or BlackHat, it simply wouldn’t be right on a main stage at DEF CON. DEF CON is, or at least it strives to be, a hacker con first and foremost.

TL;DR : This is DEF CON, please bring your “A” Game.

The Time Commitment

Often times people ask to be on the CFP Review Board because it is an honor and privilege to be among the group that selects the presentations for DEF CON… It’s also a giant time suck, which people sometimes fail to realize (or believe us when we tell them).

Now for the more formalized explanation of that so my “editor” doesn’t get pissed:

It’s been stated before, but being on the DEF CON CFP Review Board is an enormous time commitment. In the first few months, the average time a reviewer spends on talks is ten to twenty hours a week, depending on the volume of talks received. In the last two weeks, when everyone is rushing to submit before CFP closes, the time required rises to forty or more hours a week. The DEF CON CFP Review Board, like many other CFP Review Boards, is an entirely volunteer activity that many times becomes a second job. This is one of the big reasons we encourage people to submit earlier, and not wait until the last minute. Total time spent for a General Reviewer is probably in the range of 280 working hours.

The rule of the board for a General Reviewer is to do as many talks as you feel you are able to, but hit at least 70% of the talks. In practice and as far as the other general reviewers are concerned, you should be getting as close as you can to 100% of the talks. If the other reviewers feel that you’re not pulling your weight (so to speak) they will call you out. We’re like the fremen in that sense, crysknife and all. In less nerdy terms, no one wants to get shanked in the exercise yard because they didn’t review enough talks.

The topic of the exercise yard leads us into our next area, the prisons guards.. I mean, the DEF CON CFP Review Board staff.

The Defcon CFP Review Board Staff

Nikita and Alex are the foundation of the Review Process. They post the talks, interact with the submitters, deal with the reviewers when we’re cranky and obstinate (we can really be bitches sometimes), reshape the feedback given by the reviewers and transmutate those turds into flowers and candy before the submitters view it. They are the fecal alchemists and without them, the process would not work.

Similarly, there is the non-official review board staff member in the form of Jericho who tracks our submissions, votes, and other information. He categorizes the talks for us while providing amazing feedback and insight into anything vulnerability disclosure related. Like Nikita and Alex, Jericho is an integral part of making the DEF CON CFP Review Board function and prosper.

The fourth person (another unofficial one) who deserves a great amount of credit for making sure that people keep up with their reviewing is our own special CFP Vocal Antagonizer in the form of Roamer. If a review board member is slacking they can be certain that Roamer will “gently” remind them that they need to review talks. This is an important role as we want as many of the review board to provide feedback and vote on a talk as possible. This ensures more reviewers see it, and provide commentary based on their diverse background. In other words, Roamer is like a shot caller; if you don’t sack up and do the tasks assigned to you, you’re going to wake up with a horse head in your bed.  

Both Jericho and Roamer are inspiring examples of what it means to truly care about the hacker and DEF CON communities. On a personal note, it’s also pretty cool that I get to call Nikita, Jericho, and Roamer, these amazing people, my friends. I say that because after all these years, they still talk to me, even though I can be a bit dramatic.

While we’re on the topic of dramatic people, let’s talk about our special reviewers. I’m just kidding, where drama is concerned all of them pale in comparison to yours truly.    

Special Reviewers

Our special reviewers are subject matter experts who specifically comment and give their feedback on talks in their “wheelhouse”. There are many talks where the “general reviewers” simply don’t feel fully qualified enough to make the necessary judgement of a “yes” or “no” vote. Sure, they are familiar with a topic to some degree, but just don’t spend their lives immersed in that corner of security.

Everyone in InfoSec “knows” about pen-testing and social engineering for example. However, unless that is their primary tradecraft and they have been doing it for a decade or more, they may not be keeping up with the latest tools and techniques. In such cases, the general reviewers will typically “defer” to the subject matter experts. The input provided by the Special Reviewers this year has been invaluable in helping shape what DEF CON 26 will be.

Discussions

The DEF CON CFP Review Board has a unique style in how they (we) review talks in contrast to many other CFP Review Boards. There is oftentimes a lot of discussion that goes on about individual talks that plays a key part in the process. The reviewers do not live in a vacuum when reviewing the individual talks, rather, they are encouraged to communicate with one another openly on the system so as to provide a higher quality of talk selection. Sometimes the discussions may turn heated, but at the end of the day it does improve the final selection. “Heated” is a really nice term. It’s a really nice term because when we say it, you may think we might mean like a “hot summer day” when it fact we mean the fires of Mordor, or whatever is causing a burning sensation in the nether regions.

The being said, on the Review Board, it’s very important to be open to new ideas and perspectives which such discussions strongly facilitate. I don’t think the DC CFP review board would work nearly as well under any other type of system. Conversely, what works for “us” may not necessarily work as well for other CFP Review Boards.

How do I get on the CFP Review Board?

First, are you really sure you want to? Do you really have the time? The numbers we posted before about the time commitment wasn’t an attempt to oversell things (in fact they are probably conservative estimates). As a review board member you will be dedicating that much time to reviewing talks over a three to five month period, with the final weeks being absolutely brutal. And if you don’t? You’ll find yourself being called out or greenlit by a shot caller. And then the best option there is you may not be asked back the following year. Remember, you are helping to shape the tone, feel, and content of DEF CON, the longest-running hacker convention now attended by over 25,000 people. That is an incredible responsibility and you are helping ensure that attendees get value from the talks they attend.

Still want to do it though? OK. Talk to some CFP Review Board members at DEF CON 26. That’s it… just do that. Judge for yourself based on how they describe it, the good and the bad. If any of them describe a breezy stroll through a nice park with flowers and chipmunks, walk away. They aren’t telling you the whole story.

Why don’t you have a CFP Review Board Panel at Defcon?

First, it would be super boring. Invariably the attendees are going to ask us a lot of questions that we can’t answer about specific submissions. While we may “vague” tweet or generally answer a question, we can’t and won’t provide specifics on submitted talks beyond what Nikita and Alex have provided as official feedback, and then only to the person that submitted the talk. So the panel would consist of a lot of jokes, high-level “CFP tips”, and not much more value. If you really want to “know” more about the CFP, just find out where some of us hangout at DEF CON.

Before we end this first entry in this series of three or four posts. I would like to take the opportunity thank you for reading along thus far. Jericho and myself worked on this entry, but he shouldn’t be held responsible for my tangents, side notes, and improper use of some punctuation.

Credit Roll

First and Foremost, we really need to thank those people around us (friends, family, significant others) that deal with us during the three to five month a year process of reviewing talks. They truly are the unsung heroes. They know we can’t go into specifics, but they’re there to listen to us bitch and moan about “that talk”. They understand us during this endeavor when we forgo plans to hangout with them or we’re not in bed until three hours past normal time. Without their support, we could never accomplish the task laid out in front of us.

General Reviewers

Jericho Roamer HighWiz Shaggy
bcrypt Vyrus Zoz Claviger
Suggy Wiseacre Secbarbie PWCrack
KingTuna Medic Dead Addict ZFasel

Special Reviewers

Andrea Matwyshyn w0nk Malware Unicorn
Snow Kodor Grifter

DEF CON Staff

Nikita Alex

DEF CON Founder

The Dark Tangent

Shoutouts

We’d also like to give a big shout out to the Workshops Review Board. While they are a separate entity from the CFP Review Board, their contributions to DEF CON are just as important.

Tottenkoph Munin Sethalump DaKahuna
CyberSulu Kodor SinderzNAshes SinderzNAshes
Kodor SinderzNAshes Wiseacre HighWiz

In part two of the series we will be covering the statistics, because that’s the type of thing that makes some of us (but especially Jericho) super wet.

With part three will come our thoughts, and comments on the Submission Form and the Questions we ask.

Part four will be some lessons we’ve learned along the way as well as ideas for improving things in the future.

One last thing, Jericho is totally the Jimmy McNulty of the CFP Review Board.


Continue reading the second blog in this series, “DEF CON 26 CFP Basic Statistics and Observations“.

A Samsung Galaxy 8, Phantom Notifications, and @Tmobile’s Dreadful Support

This is a blog of two topics. The first, a brief technical explanation of a problem with my Samsung phone after an upgrade to Android 8.0 (Oreo) pushed by T-Mobile, the subsequent debugging, and hopefully help for anyone else experiencing the issue. The second, my horrible experience with T-Mobile Twitter-based tech support.


On April 2, T-Mobile pushed an over-the-air update for my Samsung Galaxy 8 (G8) phone. In addition to a routine Android security patch level update, it also upgraded the phone to Android version 8, code-named Oreo. Shortly after the update, I started getting what I called ‘phantom notifications’, between one and six of them every hour or less. These were audible notifications that didn’t correspond with any discernible event on the phone, sometimes in quick succession. Over the course of a week, there were a few times where an icon would appear in the notification bar for a split second, making me think it was related to a specific event, but I couldn’t figure out what. I engaged with T-Mobile on Twitter, and they offered some ideas. Here is everything I did to debug and figure this out, based on their questions and my own ideas.

  • T-Mobile: SMS App Clear Data/Cache (I suspected it may be related to SMS)
  • Me: Full power cycle
  • Me: Changed default notification to determine if the phantoms are using system notification preferences (they are)
  • T-Mobile: Verify Notification Reminder functionality = OFF
  • T-Mobile: Verify no wireless/bluetooth/NFC turned on during phantoms
  • T-Mobile: Clear cache partition on phone via Debug menu
  • Verified software versions for all functionality (‘About Device’)
  • T-Mobile: Verify all apps are updated via play store
  • T-Mobile: Verify no apps from unknown sources
  • T-Mobile: Enable Developer options (did not change anything)
  • T-Mobile: Device Maintenance showed no app crashes, no hint of a problem
  • T-Mobile / Me: Phantom notifications do NOT vibrate, while SMS is configured to (so not SMS)
  • T-Mobile: No SD card in phone
  • T-Mobile: Uninstall Samsung Health (they suspected app causing this, that app isn’t on the phone)
  • T-Mobile: Backup SMS and clear all of the messages
  • Me: DND mode suppresses the phantom notifications (observation)
  • T-Mobile: Confirm I did not download ANY new apps on Sunday (day before update), Monday (day of update), or Tue – Thur (after update)
  • T-Mobile: Confirm the last time my phone worked w/o phantom notifications was Sunday and Monday before the patch (and every day prior since buying the phone)
  • Me: twice out of hundreds of times, i have seen a ‘health monitor’ type icon appear in notifications for a split second when it happens
  • Me: One-by-one disable app notifications, wait for phantom. process of elimination = found the offending app = PROBLEM SOLVED

Naturally, it was the last app on the list I had notifications enabled for. “Weather & Clock Widget for Android” by Devexpert.NET, which worked fine on Android 7.x, started causing these phantom notifications on Android 8.0. Uninstalling and re-installing did not fix it. The only reason I had allowed notifications from this app, is it would put the current temperature in the notification bar at all times. Blocking notifications for this app didn’t allow this behavior, but also stopped the phantom notifications. No factory reset needed.


Part 2; My dreadful experience with @Tmobile tech support via Twitter DM.

First, this isn’t the first time I have Tweeted and had them reach out via DM, offering support. I don’t recall having a good experience with them before, and this time certainly takes the cake on a poor experience. I am writing this up as a warning to others who might go this route, and as feedback to T-Mobile so they better understand what it is like on the customer side, and offer some tips for improving.

Perhaps the biggest problem with T-Mobile Twitter support, is their system for interacting with customers appears to be designed to resolve issues very quickly. I can’t speak to their workload, average customer engagement time, etc. But for a case like mine? I went through 22 different people over the course of seven days. On April 8, there were nine different people that cycled through to ‘help’ me. On April 7, while working with Reggie (who happened to be the only one out of 21 that I felt was truly helpful), he said he needed to AFK for 15 minutes for break, implying that someone else would take over. By that point, I knew I had already gone through seven others, so I told him I would happily wait until he returned. This high turnover rate on support staff worked against the process entirely for my case. Each time, the new person had to try to read the thread and figure out what was going on, and they rarely skimmed the thread it seemed. When I was offered a summary of my problem by the new person, it was typically wrong or left out important bits. T-Mobile needs to better identify problems that can’t be solved in ten minutes, and keep one or a few people on the case for consistency. When a customer repeatedly asks for a specific support person to re-engage, listen to them. Here is the list of people I dealt with:

  • Apr 3 – Joel Bannister
  • Apr 3 – Harley Sumida
  • Apr 3 – Ruben Hernandez
  • Apr 3 – Dee Medina
  • Apr 3 – Zach Ricketts
  • Apr 3 – Kimmi Smith
  • Apr 3 – Victor Loya
  • Apr 7 – Reggie Reese
  • Apr 7 – Harley Sumida
  • Apr 8 – Lauren Chan
  • Apr 8 – Pete Harman
  • Apr 8 – Marva Biggar
  • Apr 8 – Sora Yi
  • Apr 8 – Marva Biggar
  • Apr 8 – Kate Tomallo
  • Apr 8 – Lauren Chan
  • Apr 8 – Meghan Parks
  • Apr 8 – Eddie Gough
  • Apr 8 – Scott Degelman
  • Apr 8 – Ray Butler
  • Apr 9 – Dee Medina
  • Apr 9 – Mike Perez
  • Apr 9 – Alex Kimbrell
  • Apr 9 – Zach Ricketts
  • Apr 10 – PoxMaphixat [1]
  • Apr 10 – Kyle Saragosa
  • Apr 10 – Scott Degelman

[1] This was the only person that didn’t appear in Twitter DMs with a real name shown by Twitter:


The next bigger problem I faced, is that T-Mobile’s documentation for their support staff is out of date. It’s as if they had never debugged an issue on a Galaxy 8, despite them selling it for half a year. During the ordeal of figuring out my problem, I ran into several times where support failed related to this:

  • Apr 3 – Document for changing SMS message sounds is outdated, not correct for G8 (you apparently can’t on this model)
  • Apr 3 – T-Mobile said to set up a notification log for debugging purposes, yet G8 removed that functionality (ridiculous)
  • Apr 7 – The location of the ‘build number’ to enter developer mode is different on the G8 than previous models
  • Apr 7 – They asked me to go to the ‘Security’ screen in options, yet on the G8 that is ‘Lock Screen and Security’
  • Apr 7 – T-Mobile diagnostic data said ‘apps from unknown sources’ was enabled, my screen said it was disabled
  • Apr 8 – They asked me to check the ‘Samsung Health’ app (there is none, apparently part of the ‘Activity Zone’ app, but that function is disabled)
  • Apr 9 – T-Mobile kept telling me a factory reset is the way to fix this, despite it not necessarily working
  • Apr 10 – T-Mobile told me a factory reset is the way to go AFTER I solved the problem (WTF?!)

After having to correct the T-Mobile support staff this many times, and figure out how to find what they were looking for, it shows an obvious gap in their support ability. As someone who wrote my fair share of technical documentation, I cannot stress how important this is.

As mentioned above, when a new support person steps in, they have to skim the thread to catch up. One person told me that they take extensive notes to alleviate that problem, but after most of the new people offering me a summary got major parts wrong, I don’t think that is the case. Even if they do take notes, I think they are not consolidated, not done in a way for easy transition of the case, and generally convoluted. This causes the support staff to repeat the same things, ask the same questions, and waste customer time.

Next, T-Mobile needs to make sure their employees understand policy. Compare:

  • Apr 3 (Vinny) – “Thanks a bunch for remaining engaged with us at T-Force today, my name is Vinny and I’ll be taking over from here, as Krystn, as she had to step away.”
  • Apr 3 (Joel) – “Thank you so much for reaching out to T-Force! My name is Joel and I will be your #MagentaExpert!”
  • Apr 3 (Ruben) – “I hope you are having an amazing day. My name is Ruben and I will be taking excellent care of you and all of your concerns/questions today.”
  • Apr 3 (Zach) – “Thanks for sticking with us here. My name is Zach, and I’ll be taking over from here.”
  • Apr 7 (Reggie) – “I do want to introduce myself, my name is Reggie and I will be your #MagentaExpert today.”
  • Apr 8 (Meghan) – “My teammate had to step out for a quick meeting but my name is Meghan and I’ll be taking over to provide you with excellent service!”
  • Apr 8 (Eddie) – “Fun fact, Since T-Force is a team and constantly changes to ensure that customers always have support 24/7 we are not supposed to share our name since it already shows on the message.”

After support staff introduced themselves by name six times, Eddie came along and said they aren’t supposed to share their name. He further points out that Twitter shows their name (in the native web interface, not in Tweetdeck BTW), and yet that isn’t the case either as seen by “PoxMaphixat” above.

While some that interact with T-Mobile may say they are really ‘nice’, to me, that isn’t the case. Their overboard attempts to portray a fun and friendly atmosphere are insulting and a waste of time. Throughout the week, I was assured that they were there to help and resolve my issue, while not reading the prior messages, not understanding the issue, and bouncing in and out of my ticket to the point it was difficult keeping up with them. The phrase they loved to over-use, “I will be your #MagentaExpert!” is a joke. Seven days to figure out my problem, and they never did, I had to. Other phrases they love to say, adding fluff and not actual support, while not reading the thread and repeating the same things over and over:

  • I absolutely want to be able to help you in any way that I can!
  • It’s great seeing you here today. I hope you are having an amazing day.
  • That is an awesome question and definitely not something I am familiar with, but we can definitely work together to look into it!
  • I honestly want the best and fastest resolution for you!
  • Thank you for taking time out of your day on this!
  • Here at T-Force, we value customers time and always want to get them the best resolution possible without wasting their time.
  • We’ve got your back! (T-Mobile needs to remove this from their playbook, it is insulting.)
  • I really appreciate you reaching out and working with T-Force today.

Overall, I need a lot less of this fluffy wording, and a lot more I didn’t quote, and more actual support. If you have to keep telling me you “have my back” and want to give me the “best resolution possible”, you are convincing me you aren’t good at your job. We expect customer support to do that already.

Apr 3 (Joel) – “If you prefer to not do that, then you always have the option to back up the device and reset the software completely.”
Apr 3 (Zach) – “Can you please tell me if you’ve completed a master reset on the device since the update?”
Apr 3 (me) – “If a ‘master rest’ means a ‘factory reset’, that may be a deal breaker.”
Apr 3 (Zach) – “Typically, if there are any bugs that come across after an update, which this one may just be, a factory reset would be the best possible solution, as inconvenient as it can be to set everything up again.”
Apr 3 (Kimmi) – “In those instances the only fix I’ve been able to locate based on user feedback is a factory reset of the device.”
Apr 3 (Kimmi) – “Unfortunately the only option we have at this time is to complete the reset.”
Apr 3 (Victor) – “The master reset would be a great way to fix the issue in case it’s just some sort of temporary issue. ”
Apr 7 (Reggie) – “By no means do I want to tell you that you absolutely must do this, but in the end I want to respect your time and I feel like at this point the Master reset might fix the issue permanently whereas what we have done has demonstrably had no effect on the issue at hand.”
Apr 7 (me) – “If a factory reset is the answer, then I walk from Tmobile and go on a social media campaign to dissuade people from using Tmobile, because that is just sloppy programming and a complete breakdown of tech / customer support.”
Apr 8 (Marva) – “I know Reggie mentioned a master reset and that seems to be the only thing we haven’t tried up until this point, is that correct?”
Apr 8 (me) – “Safe mode has not been tried, and a reset, the nuclear option, is out of the question.”
Apr 8 (Sora) – ” I know that you do not want to do a master reset … I totally follow your logic; I do want to mention that if the software update is giving this error, then a master reset does allow the software to be restored on your phone properly.”
Apr 8 (Marva) – “The next step in troubleshooting is to complete that master reset.”
Apr 8 (Kate) – “The Master Reset sounds nuclear, but truly is the faster and cleanest resolution available.”
Apr 8 (me) – “As I said earlier this week, a factory reset means I will no longer be a T-Mobile customer, and will blog about this entire mess, that T-Mobile sent faulty software and could not debug it, and now is pressuring me to go that route while ignoring my direct questions about Samsung Health buginess, that icon that shows sometimes, and my desire to explore that route. That said, do you still think a factory reset is the right option instead of pursuing valid leads that may fix this without a reset?”
Apr 8 (me) – “From there, process of elimination can tell likely tell us which app is causing them. No safe mode, no factory reset. Please add this to your CS playbook.”
Apr 8 (Eddie) – “With the awesome software that we have nowadays, a master reset is the best option since there’s a high chance the bug will be deleted, and your information will be downloaded onto your phone within less than one hour if it’s backed up”
Apr 8 (me) – “Ugh, STOP. Do not recommend a factory reset to me again. I just gave a viable option to better figure this out that will take a few hours, and you go back to factory reset, after I have REPEATEDLY said that is a nuclear option and I a) will not do it OR b) do it and no longer be a tmobile customer.”
Apr 8 (Eddie) – “I just wanted to assure you that we are going to be here for you until we get a resolution. Never wanted to tell you that you should do a master reset.”
Apr 8 (me) – “I mentioned I found a new solution to this kind of problem, to add to your play book. And you immediately recommend a factory reset despite me REPEATEDLY saying ‘no’. You understand no means no right? I am tired of being told why a master reset is the option, and I am *more* tired of Tmobile reps not reading why it is NOT necessarily the right option, why it is NOT a guarantee it will fix anything.”
Apr 9 (Alex) – “If so, have you installed them and reinstalled them? Those are the first two steps, so let me know how that goes!”
Apr 9 (me) – “Two? There were *19* people on the Tmobile side during the course of this investigation, all of who gave up and told me to factory reset.”
Apr 9 (Alex) – “Now, I know we mentioned a master reset was something we should try.”
Apr 9 (me) – “Pretty much confirmed, “Weather & Clock Widget for Android” by http://Devexpert.NET is the one causing the phantom notifications. Uninstalling and re-installing it to start.”
Apr 9 (me) – “Uninstall & Reinstall did not fix it. So there is some weird issue between the app and the Oreo update. I can get around this by disabling notifications for that app, which only makes it so I don’t get the temperature in my notification bar. With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “I also explicitly said last night to STOP telling me to factory reset.”
Apr 9 (me) – “I have asked half a dozen times and every single one of you jerks ignore me. Focus on THAT problem instead of a factory reset.”
Apr 9 (me) – “With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “At this point i am 99.99% sure I have this resolved, again, without a factory reset.”
Apr 10 (PoxMaphixat) – “Resetting the device and processing a warranty exchange is our last resort. Which would result in a device that is fully reset as well. This might be the thing we would need to do since we’re not able to resolve this phantom issue.”
Apr 10 (me) – “Not only have i solved the issue, I have said repeatedly NOT to recommend a factory reset to me, and you assholes keep doing it. NO MEANS NO.”
Apr 10 (Kyle) – “We can see that you’ve invested a lot of time with these issues on your phone and wanted to avoid going through the previous steps that’s you’ve already done, which is why we were looking at the master reset as a last resort … So our troubleshooting steps would basically be the master reset as well though I Samsung may have more support on what’s going on with this app.”
Apr 10 (me) – “Seriously? You suggest a master reset AGAIN when I have said over and over NOT to tell me that? I solved the phantom notification issue without a reset,”
Apr 10 (Kyle) – “I would reach out to Samsung as I completely understand your concern regarding the reset and they would be able to support the app even further. Does this make sense, Brian?”
Apr 10 (me) – “You said ‘reset’ again. How can I be any more clear here? Never, EVER, not a single time, EVER tell me to factory reset my device. Don’t even mention the word ‘reset’, let alone ‘master reset’ or ‘factory reset’. I honestly feel like there is a den of rapists and molesters working at Tmobile, who don’t understand what the word ‘NO’ means. Does this make sense, Joel / Harley / Ruben / Dee / Zach / Kimmi / Victor / Lauren / Pete / Marva / Kate / Meghan / Eddie / Scott / Ray / Mike / Alex / Zach / PoxMaphixat / Kyle?”

After this? Scott said ‘reset’ once more shortly after my last message. This is the text-book definition of the worst customer support that can be offered. A customer specifically says, over and over, not to recommend a bad support option (the factory reset). Yet, T-Mobile kept recommending it every single time. It gets to the point where it is a trigger word for me, because it clearly shows the support person didn’t read the prior messages. It means that the support staff didn’t leave a message for the next person not to bring up a factory reset. Worse? I SOLVED the technical issue, without a factory reset, and said as much. T-Mobile’s solution? Keep recommending a factory reset anyway, when it was clearly not needed. This is hands-down the worst customer service you could possibly offer, and completely insensitive to a customer. I don’t really care where the breakdown happened, other than it happened half a dozen times, but when a customer says “do not do $thing“, you should NOT do $thing. No questions, no arguments, no equivocation. Yet T-Mobile ignored that basic point, that basic understanding of the tenets of customer support. 18 separate times, reset was their answer, three times after resolving my issue.

My next advice for T-Mobile is to embrace an old classic of customer service. Over six days, interacting with 21 different support people, after repeated complaints about many of them, no manager stepped in. At least, no one identified themselves as a manager, no one exhibited any signs they were a manager, and absolutely no one made it a point to get me a resolution other than the empty “we’ve got your back” lies. Imagine going into a Taco Bell and talking with 21 employees trying to resolve a problem, that your Mexican Pizza was missing ingredients or not cooked, and that entire time no manager stepping in to ensure you got a properly prepared and cooked food item. To me, the customer, those scenarios are no different.

Finally, the bigger picture. I engaged support for one problem, the phantom notifications, which I eventually resolved myself. During the process, T-Mobile asked me questions that highlighted other problems. Despite figuring out the original, I left the engagement with two additional problems that they did not resolve. First, I asked how to disabled ‘Bixby’ completely, and they couldn’t help. Like so many other things, they didn’t understand the software, and/or their documentation wasn’t updated. I had to tell them to disable it per their instructions, it required creating a Samsung account. You actually can’t access the real settings of that malware without creating an account. That is atrocious and just bad design. Second, when we went down the road of the occasional phantom notification icon that I saw, it led us to the ‘Samsung Health’ feature within ‘Activity Zone’. On my phone, it says “tap here to get started” and tapping there does nothing. T-Mobile never helped with that, and after specifically asking them to half a dozen times, they told me to talk to Samsung.

Two more bonus observations, that came up during this ordeal. First, the T-Mobile software update downloaded over 4G, not WiFi. It used to prompt you if you wanted to wait for WiFi and this time it did not. Second, I mentioned that T-Mobile was still sending SMS notifications to me before 9 AM, and one of the support people were gung-ho saying that was not right, they would take my complaint to the top! Well, good luck there, since the last time I brought that issue up on Twitter it did go to the top, all the way to the office of the executives. Nothing ever came of it and I still get text messages from them before 9 AM. If you are going to grab that flag and head on a crusade on my behalf? Maybe consider better helping fix my original problem first.

So, T-Mobile, I have given you a wide variety of ideas for improving customer support. It is in the context of a support case you can easily reference. These ideas are very much in line with many other support services offered by similar services and companies. It’s time for you to up your game.