The Attrition DC26 Badge Challenge Post Mortem

This year, which was my final trip to DEF CON, I made up one last round of Attrition DEF CON badges. In prior years they were typically engraved luggage tags a bit more specific to the year:

Since #BadgeLife has become a big thing, especially this year as far as I can tell, I decided to go a bit lower rent on the badge material but ‘up the game’ on the content. I did a ‘cipher challenge’, which of course was never meant to be a real challenge. I’m not nearly smart enough for that shit. I literally came up with it in less than a day, didn’t vet it with anyone, and just moved to mock up a badge and print. Because I am so pro! I also figured anyone who knows me would know not to trust me on anything ‘cipher’ or ‘challenge’, especially ‘cipher challenge’. Unfortunately, and I do feel bad, a handful of badge-holders went down this rabbit hole.

This write-up is for them, to explain just how fast this was put together, and the lessons I learned as well. The cliff notes details, as I originally intended:

  1. https://en.wikipedia.org/wiki/Cirth (hobbit) -> “never trust us”
  2. https://en.wikipedia.org/wiki/Wingdings -> “except this time”
  3. location hint (flamingo hotel) -> “Phoenicopteriformes”
  4. refined location – wildlife habitat long/lat -> 36.11662720392657 / -115.17115294683322
  5. 08/11/2018 @ 3:04am (UTC) Epoch Unix Time -> “1533956647”
  6. Klingon “take proof you were there” -> “pa’ SoH’a’ tob tlhap”
  7. random letters/numbers -> (unsolvable/gibberish)
  8. show Jericho proof (latin) -> ostende inamabilis sciurus
  9. winner winner chicken dinner -> (icons)

Seems pretty straight forward! Unfortunately, a few of these didn’t work out so well as I found out, in surprising ways. Here are the hiccups I didn’t expect.

  • (1) There are multiple Cirth character sets. Pretty minor, but it led to a couple people saying the translation was off. Worse? That one character that was off fed into another hint and made it more believable. I should have read through the Wikipedia article to notice that, but growing up as a skilled writer in ‘Tolkein Runic’ (Cirth), I didn’t think about it.
  • (2) Always trust the first hint, never the second!
  • (5) So… Epoch Unix Time is an absolute. You don’t adjust for timezones, because the time is in Coordinated Universal Time (UTC). The Wikipedia entry for UTC confirms it “is not adjusted for daylight saving time“. So my intention of it being on Saturday morning at 3:04am was correct. I didn’t account for everyone adjusting for time zones. I also didn’t account for some adjusting for Las Vegas’ time zone (Pacific) or trying to second-guess it and using my time zone (Mountain). At this point I am vindicated, anyone loitering around flamingos at the Flamingo between ~ 8p – 10p local time, were not following the cipher. Yes, I still feel bad they showed up thinking there was a prize/reward there.
  • (6) I really should have known better here, since Google Translate fails to translate simple text from one language to another, and then back again. I fell to this trap using the first Klingon translator that Google offered and did a simple one-way translation. Unfortunately, that same site changed “take proof you were there” drastically to involve something with a cat in it. I like cats, everyone knows this, so the clue still had some crazy merit. Fortunately for me, one of the badge-holders knows a lot more about Klingon than the online translators do, and gave me a deserved verbal beratement over the horrible translation. This led me back to that translator, where I pasted “pa’ SoH’a’ tob tlhap” back into it and got, you guessed it… “you take a cat room“. This was a solid break in the intended chain, and a deal breaker for solving the badge. Oops.
  • (7) This line had a simple intention. This line may have been the weirdest in the long run. A bunch of random numbers and letters, with no intended meaning, to be an ultimate ‘gotcha’. So no one could say they solved it, or if they did, I could challenge them on that line. I left this up to the wonderful badge artist, Anushika, who typed in a random string while designing it. Between that and the chosen font, there was even question over one or two characters. Either way, I thought it served a purpose. One nice lady from Australia (she is nice, despite her DMs irrationally suggesting I not to call her that) spent a lot of time on this, maybe more than anyone else. At one point she messaged “Threw it through successive shifts. And the answer it gave me was successive shifts.” This was after I reminded her on previous comments, that “i’m not really bright. hashed, encrypted, encoded… i get so confused“. No false modesty or deception; math is a religion, and I don’t believe. Ergo, crypto is a foreign language to me for the most part. So that random line had some merit in the math world maybe? Put it through successive shifts, and the answer is more successive shifts. That certainly sounds like I was really brilliant in a troll cipher, when I was the farthest thing from it. She kind of spooked me when she told me that and I thought “oh shit, this line has meaning?!” Kind of disappointed that a ‘troll cipher’ isn’t a real thing with a Wikipedia entry!
  • (8) Translation woes again. As someone who took a year of Latin in high school, seriously, and knows about the headache of online translators… not sure how I got burned twice in one badge. I translated “show squirrel proof” since I knew it wouldn’t handle “jericho”, and got “ostende inamabilis sciurus“. This is where it gets really weird. Someone messaged while in Vegas that the translation was off, and I went to check again, using Google Translate again. Click that link and you will see the problem. The translation changed between making the badge, and someone translating it after receiving the badge, which was around 30 – 40 days. So now it became “inamabilis sciurus ostendit probationem“. This caused a problem because the first translation now reverses as “show squirrel” which is lacking a crucial word. The updated translation, when reversed, comes back as “squirrel proof shows“, which is a bit closer to the intent. Ugh. For fun, since we had to pick ‘Latin’ nicknames in my Latin class, I chose Sylvester. #JerichoTrivia

So there you go badge-holders and adventure-seekers! I sincerely apologize for any hardship you went through, to a degree, because that first line really is gospel when it comes to me, attrition, and anything remotely close to a challenge. Years prior, I wanted to do a luggage tag badge like those pictured above, but cut out holes in a Goonies sort of way along with instructions to stand in the middle of Las Vegas Blvd to line up three landmarks to figure out where the party was. After this badge challenge? Probably for the best I didn’t, or I bet I would have gotten a few people run over. On the upside, you got to spend time with Flamingos, largely more bearable than the average DEF CON attendee.

Advertisements

The Uncertain Future of Necco Wafers and the Logical Response

Recently, the Necco wafer factory abruptly shut down after the company sold it to an “unknown buyer”.

The footer to that image reads: “Necco, the oldest candy company in the country, abruptly shut down its Revere, Mass. factory on July 26, and left about 230 workers jobless. (Reuters)”

Yes, the oldest candy company in the country! This is history right here. We must preserve and honor it, do everything we can to preserve it, even if a tiny majority of Americans enjoy Necco wafers (like me)! I’m not the only one… Newsweek reports, “Fans stock up as America’s Oldest Candy Company Faces Closure”.

I caught wind of this several months ago, and as a fan of Necco wafers, I was obviously worried. So I did what any red-blooded, patriotic, Type-1 diabetic American would do… I bought some.

I bought 154 rolls of Necco wafers, including the rare Sour ones that are doubly delicious.

That is 33,850 calories of Necco wafers.

That is 8,624 carbohydrates (sugar) of Necco wafers.

And my insurance provider tried to tell me and my doctor that I didn’t need insulin as a Type-1 diabetic. CHALLENGE ACCEPTED.

Jericho in Vegas Next Week… (for real)

Hi!

Given my occasional good-natured trolling on Twitter, and since many have asked me the last few weeks, I want to set the record straight. I will be in Las Vegas next week, for real. I arrive tomorrow evening and leave the following Sunday. This is the first time at BH/DC in several years for me.

Between Monday and Wednesday I will be doing the corporate thing around Mandalay and adjacent to the Black Hat event. I am not actually attending the conference, thus ‘adjacent’. Each day already has several meetings lined up so I won’t be readily available for parts of the day. When not in a meeting, happy to meet up with anyone looking to better understand the nuances of the vulnerability intelligence landscape. On Tuesday evening I will be at the Guidepoint Party at the Aureole in Mandalay Bay for several hours. Wednesday night I hope to crash the BSidesLV pool party and enjoy the cool 94 degree temperatures Vegas has to offer at night.

Between Thursday and Sunday I will be doing the hallway thing at DEF CON primarily. On Thursday at 3:30p I will be on the DC101 panel, apparently because I am old, to dish out horror stories about our industry to those attending. On Friday and Saturday I will no doubt be around Skytalks on and off to harass and support that track. Otherwise, you can likely find me roaming around Caesar’s and Flamingo checking out villages and side events.

I have a Twitter client on my phone but it doesn’t have any alerts, so that won’t be a reliable way to reach me. I hope to check Twitter every so often but my lizard brain isn’t wired to check that really. If I do camp down at a spot in a hallway or bar I hope to remember to Tweet my location in case anyone wants to discuss wildlife rehabilitation or vulnerability databases or anything else interesting really. As for spotting me, I will be one of ~ 100 wearing the DC26 Attrition badge, and a T-shirt that has an animal on it. As many have said, I too am really bad at remembering names while fairly good at remembering faces. Worse, when I do remember trying to figure out if you prefer to go by real name or handle at what events. Please don’t be offended and please re-introduce yourself! It may take me a minute to remember our history, my brain is a tad broken these days.

Finally, this will be my last year attending DEF CON. I attended DEF CON 2 back in 1994 at the Sahara, so this will be my 25th anniversary. I see a lot of value in DEF CON and continue to volunteer reviewing talks on the CFP panel to help shape the conference and try to make the content the best possible. Next year I will stay on with CFP in a more limited role, but still offer my input for certain types of talks. That said, as many say before and after ‘hacker summer camp’, the week is emotionally and physically draining, and many of us often come back with ‘con flu’ or some other kind of crud. The last time I attended, I went a full week not seeing some friends that were in Las Vegas, because the meta-convention is just so big and spread out. I hope that doesn’t happen this year, but it is one discouraging aspect of a week in Vegas.

While DEF CON doesn’t work so well for me personally, I see a lot of potential in it especially with the huge rise of villages. More and more that I talk to say that the villages are the first part of the conference that attracts them, more so than the main lineup of talks. Villages are an evolved modern evolution of old “birds of a feather” sessions at conferences back in the day, before ‘hallway con’ was a thing even. A group of people that share a particular interest and want to focus on a given topic have the ability to do it. Even better, often times that comes with elaborate and painstakingly designed networks and challenges to test your skills and learn more. In addition to villages are the side events for runners, shooters, coffee-drinkers, and more. I encourage everyone, especially newcomers, to embrace these side events and villages. DEF CON will be what you make of it, and there is more opportunity now than ever before to make the best of it.

DC26 Attrition Badge Round-up

This is the first DEF CON I am attending after a long break. For kicks I decided to make up a run of DC26 Attrition badges like prior years and conferences. Depending on who you ask, the badge is a decoration only, or it gets you into fabulous parties and amazing events. Anyone with a badge is encouraged to embellish.

Since the July 5 announcement of the badge, I increasingly focused on using them to raise money for charity. That, in turn, prompted several people to ask for details of the badges and the money raised. This blog will hopefully answer those questions and maybe inspire others to help out when they can. If you aren’t interested in the quick story, scroll down to the inspiration section please.

First, a link-heavy summary. On July 7, I did the first charity challenge looking to raise money for the ACLU, GLBT Community Center of Colorado (The Center), and Planned Parenthood. I also started giving out a a handful of personal challenges to random people expressing interest in a badge with fun results.

On July 10, I did a second charity drive bigger than the first. I also offered one badge up as part of an art challenge for the best original art featuring Lazlo. Deathjaw17 won that with this epic piece:

In addition to the art, I did a few other trades including for this slick challenge coin as well as a few other DC26 badges. At this point some of the winners of badges started posting pics, including with chickens, with epic beasts, and with bubbly! The Lazlo badge also got a tour of Philly and a sweet visit to the CompSci building in War Games. One badge went out and lead to a fun picture and backstory of a ‘dojo squirrel’. During this process, I got an unexpected care package from Kentaro, that he sent before I sent him a badge, and @Otterannihilation received a badge and sent back an amazing gift as a thanks. Meanwhile, pictures of badges kept coming:

    Inspiration and the Opposite

By this point, after two big charity drives, and several subsequent one-off drives, it was clear to me that raising money for charities was a great option. Badges were in demand and a lot of great people were willing to throw in money to help great causes. This also led to some other great opportunities that aren’t donations to charity, but amazing ways to help out. The level of inspiration and good-will in our industry is always refreshing, one of the few things that keep some of us from losing all hope. More on that later.

The opposite of inspiration came in two forms. First, while the badges w/ lanyards cost $298.60, but the postage to mail them out to x people cost $448.12, meaning the entire effort cost $746.72. This was due to the lanyards, which meant the badge couldn’t go as an envelope; they had to go as a package. Each envelope cost $3.50 domestic, $10 Canadian, and between $13.75 and $14.25 to mail international. This resulted in one fun trip to the post office that took around 30 minutes and produced a generous receipt.

The second came in the form of being questioned and challenged about my badges repeatedly, and being accused ofstrongly [reinforcing] exclusive cliques within infosec“. After assuring someone this was not “a dark stunt satirizing infosec exclusionism and signaling“, giving information on the charity contributions at the time, and reminding everyone that “the charity-driven badges are open to *anyone*. i have sent badges last week, and will send some this week, to people I don’t know and have had little to no interaction with“, I still faced questions about if I was reinforcing the exclusive cliques in infosec. I’ll say this definitively; I am not reinforcing cliques at all. This is trivial to see if you remember the definition of a ‘clique’, and consider that I don’t know half the people getting a badge other than a brief Twitter interaction.

OK, back to the inspiration. At the suggestion of Noah, with his input, two badges were given out to people who volunteered to provide InfoSec training for free. First, Jim Manico volunteered to give one of his well-known and appreciated AppSec classes in December on his birthday, for free, with the focus of recruiting women, LGBQT, and/or PoC for the class. Additionally, Bones volunteered to give design and give an infrastructure/cloud security pentesting course. I also suckered her into slipping in a not-so-subtle requirement.

An even bigger inspiration, and one that shocked me, was the community stepping up to donate to charity for a badge. Once I saw the generosity, I ran with it and focused on using a majority of the badges to continue raising money for charities I support, and ones that the donors support. The charities that received donations in return for badges included the ACLU, Cavy Care, Center for Genocide Research and Education, Colorado Animal Rescue, Electronic Frontier Foundation, Greenwood Wildlife Rehabilitation Center, Hawaiian Humane Society, Kids in Need Foundation, Planned Parenthood, Retriever Rescue of Colorado, SaveABunny, Special Operations Warrior Foundation, Sprout Therapeutic Riding and Education Center, The Wild Animal Sanctuary, and Women in Security and Privacy (WISP). A total of 69 donations from 67 heroes between 2018-07-06 and 2018-07-28, raised a total of $8453.47. I’m still happily shocked at this outcome.

I also want to thank Heidi for chatting and educating me about Women in Security and Privacy (WISP) and their initiative to help more women get to DEF CON. Over a week of chatting, it started out as “this is my first DEF CON and it is rough financially” to her being one of the recipients of the WISP grants. Even better, one of the people that donated and won a badge said to give it to someone else. I suggested Heidi and they said that was a good choice! So on top of getting help to DEF CON, she got a badge, and I threw in some stickers to round out the fun.

Finally… are you sad you didn’t get a badge? Depressed that you didn’t get a chance to donate to charity to win one? Fortunately for you, there is one last chance! Jives reached out and we’re partnering for a big charity auction, with a couple days left! You can bid to win a DerbyCon ticket, a DC26 Attrition badge, and a custom box of shit! Bid now, bid often, win this sucker

EFF Lock Screen Graphics – FYI and a Minor Touch-up to One

For those who haven’t seen, the Electronic Frontier Foundation (EFF) has created several lock-screen / wallpaper images related to protecting your rights. I wanted to use the first one on my Galaxy S8 Active, but the image interferes with seeing the clock, date, and notification icons. So I moved the text of the image down just enough so that it fits comfortably while not obscuring any information. Screenshot below, and a link to my version of the image you can download.

DEF CON 26 CFP Basic Statistics and Observations

This is the second blog in a series about DEF CON 26 CFP. The first:

A Look Into the DEF CON CFP Review Board (we’re actually really boring people)


First, this post is not sanctioned by DEF CON in any way. I am a member of the CFP team who decided to keep some rudimentary statistics on the submissions this year, as I did last year. I did this to give the team a feel for just how many submissions we got, how many talks we accepted, and primarily to track the way we voted. This greatly assists the powers that be (the amazing Nikita) to more quickly determine which talks are well-received. Due to time constraints, I was not able to track as much metadata, so this blog will be shorter than last years.

First, a few bits of information:

  • DEF CON 26 CFP opened on January 16, 2018
  • DEF CON 26 CFP closed on May 01, 2018
  • Two talks were submitted after closing date and were considered for various reasons
  • We received 551 submissions (up from 536 last year)
  • Four of the submissions were withdrawn by the submitters by the end of CFP
  • BlackHat received around 1,000 submissions this year for comparison

A recurring theme in these blogs and our Tweets throughout the CFP process is strong encouragement to submit early. While we did get a share of submissions in January and February, you can still the huge spike we experience in April (a majority a day before CFP closed), and May (on the day it closed). The two weeks between the end of CFP and the time when acceptance/rejection letters are sent out become stressful as we’re under deadline to review talks, try to get last minute feedback when we can, and make final decisions.

Of the 551 submissions, 107 were accepted (19.4%). There were 388 unique male submitters, 39 unique female submitters, and 14 anonymous submissions (note: we only catalog based on the gender, if known, of the primary speaker). Of those 14 anonymous submissions, 3 were trivially identified because the submitter didn’t scrub their submission properly or submitted work that had been presented before and was caught with a quick Google or Bing search.

Of the 551 submissions, 173 (31.40%) said they would release a new tool. 77 (13.97%) said they would release an exploit, up from 56 (10.53%) last year. Of all the submissions, 216 (39.20%) were also submitted to Black Hat and 51 (9.26%) said that speaking at DEF CON was contingent upon Black Hat accepting their talk. Only 73 (13.25%) submissions were also submitted to BSidesLV. Of the 551 submissions, 122 of the speakers had presented before at DEF CON, and an additional 28 had presented before at a DC Village or Workshop.

Unfortunately, time did not permit me to properly track ‘red’ vs ‘blue’ vs ‘black’ submissions, nor categorize the talks. That said, 11 talks were about ‘Artificial Intelligence’ and/or ‘Machine Learning’, even if some of them didn’t quite seem to know what those terms really mean. Ten submissions were on the topic of, or heavily related to, blockchain. Eight submissions came with the ultra creative title that included “for fun and profit”, four included “all your $blah belong to us”, two submissions used “pwned” in the title, and fortunately for our sanity, none wanted to make $blah great again.


That’s it! I realized this is a bit more brief than last year, but the time requirement of reviewing all of the submissions is crazy. Finding extra time to maintain the sheet is rough, and generating even more statistics or tracking additional metadata just can’t happen sometimes. Fortunately for me, this year Highwiz stepped up and did an incredible amount of work filling in data, especially while I was lost in the mountains for a few days. 

A Look Into the DEF CON CFP Review Board (we’re actually really boring people)

Written by Highwiz with contributions and editing from Jericho

Being on the DEF CON CFP Review Board can be as exciting as {something}; as frustrating as {something}; as thought provoking as {something}; and as enriching as {something}. It’s like mad libs, I hope you’ve filled in this section with something good.

Each year, myself and somewhere between 16 and 20 other reviewers take on the responsibility of selecting the best possible talks for DEF CON.

Oh, I should also apologize in advance as you read this first entry in the CFP Blog series. I apologize because I am not known for my brevity. In the “written word” and especially when it comes to something I’m passionate about, I tend to be wordy AF. [See, like that sentence: Could have just said “Hope you enjoy”, but nope – not me…].

I do genuinely hope that someone finds these blog postings helpful and that it will allow submitters (or potential submitters) some insight into the way we work so as to better prepare their submissions in the future.

In its original form, this post was about as dry as some of the white papers we read that were included in several submissions. Speaking of, white papers help tremendously when we’re reviewing your submissions, and if you include one, you’re already ahead of the pack. Sadly however, while White Papers do indeed help your chances during the CFP, they make for really shitty blog posts.

While we’re on this wildtangent of things that are related to the CFP Board but not actually part of the CFP Process itself, let’s talk about the term “CFP”. Above, I mentioned white papers; while the term CFP originally did mean “Call For Papers”, it doesn’t anymore. Most people don’t submit papers. When you think about the term CFP, you should really think of it as Call For Presentations. I know I’m not the first person to say that and I definitely won’t be the last, but still, it bears saying.

Alright, back to the topic at hand…

This year, the DEF CON Call for Presentations (CFP) Review board was made up of 16 “General Reviewers”,  six “Special Reviewers”, and two members of the DEF CON staff.

The DC CFP process is not “blind”, meaning reviewers can see each other’s votes, and we see who submitted it unless they specifically opt to stay anonymous (and properly scrub their submission). There are merits for both open review and blind review, but we’ve found that an open review significantly helps our process as there is a lot of good discussion about each individual submission. One reviewer may spend considerable time digging into the topic, researching prior disclosures or talks along the same lines, or offer their personal in-depth knowledge which typically helps several others better understand the topic and state of research.

If you submitted a talk to DEF CON this year, then all of the General Reviewers most likely reviewed and discussed your talk. While these reviewers tend to agree on many talks there are also submissions that cause arguments and intense heated discussions. Most of the review board members have a very extensive vocabulary and seem to enjoy finding new and creative ways to use the word “fuck” in a sentence (both in the positive and negative). Though, why the topic of vocabulary is at hand, let me say this to my fellow review board members: y’all motherfuckers need to find a new word besides “pedestrian“. I’ll leave it at that.

As reviewers, every year we’re often left wondering why certain people have chosen to submit to DEF CON and whether or not they actually understand what type of conference it is. A prevailing sentiment on many submissions is “This is not a DEF CON talk”. While the content may be of significant quality, the question we often ask ourselves is “is this talk right for DEF CON?”. Sometimes the answer is that while it would be good at a developer conference, RSA, or BlackHat, it simply wouldn’t be right on a main stage at DEF CON. DEF CON is, or at least it strives to be, a hacker con first and foremost.

TL;DR : This is DEF CON, please bring your “A” Game.

The Time Commitment

Often times people ask to be on the CFP Review Board because it is an honor and privilege to be among the group that selects the presentations for DEF CON… It’s also a giant time suck, which people sometimes fail to realize (or believe us when we tell them).

Now for the more formalized explanation of that so my “editor” doesn’t get pissed:

It’s been stated before, but being on the DEF CON CFP Review Board is an enormous time commitment. In the first few months, the average time a reviewer spends on talks is ten to twenty hours a week, depending on the volume of talks received. In the last two weeks, when everyone is rushing to submit before CFP closes, the time required rises to forty or more hours a week. The DEF CON CFP Review Board, like many other CFP Review Boards, is an entirely volunteer activity that many times becomes a second job. This is one of the big reasons we encourage people to submit earlier, and not wait until the last minute. Total time spent for a General Reviewer is probably in the range of 280 working hours.

The rule of the board for a General Reviewer is to do as many talks as you feel you are able to, but hit at least 70% of the talks. In practice and as far as the other general reviewers are concerned, you should be getting as close as you can to 100% of the talks. If the other reviewers feel that you’re not pulling your weight (so to speak) they will call you out. We’re like the fremen in that sense, crysknife and all. In less nerdy terms, no one wants to get shanked in the exercise yard because they didn’t review enough talks.

The topic of the exercise yard leads us into our next area, the prisons guards.. I mean, the DEF CON CFP Review Board staff.

The Defcon CFP Review Board Staff

Nikita and Alex are the foundation of the Review Process. They post the talks, interact with the submitters, deal with the reviewers when we’re cranky and obstinate (we can really be bitches sometimes), reshape the feedback given by the reviewers and transmutate those turds into flowers and candy before the submitters view it. They are the fecal alchemists and without them, the process would not work.

Similarly, there is the non-official review board staff member in the form of Jericho who tracks our submissions, votes, and other information. He categorizes the talks for us while providing amazing feedback and insight into anything vulnerability disclosure related. Like Nikita and Alex, Jericho is an integral part of making the DEF CON CFP Review Board function and prosper.

The fourth person (another unofficial one) who deserves a great amount of credit for making sure that people keep up with their reviewing is our own special CFP Vocal Antagonizer in the form of Roamer. If a review board member is slacking they can be certain that Roamer will “gently” remind them that they need to review talks. This is an important role as we want as many of the review board to provide feedback and vote on a talk as possible. This ensures more reviewers see it, and provide commentary based on their diverse background. In other words, Roamer is like a shot caller; if you don’t sack up and do the tasks assigned to you, you’re going to wake up with a horse head in your bed.  

Both Jericho and Roamer are inspiring examples of what it means to truly care about the hacker and DEF CON communities. On a personal note, it’s also pretty cool that I get to call Nikita, Jericho, and Roamer, these amazing people, my friends. I say that because after all these years, they still talk to me, even though I can be a bit dramatic.

While we’re on the topic of dramatic people, let’s talk about our special reviewers. I’m just kidding, where drama is concerned all of them pale in comparison to yours truly.    

Special Reviewers

Our special reviewers are subject matter experts who specifically comment and give their feedback on talks in their “wheelhouse”. There are many talks where the “general reviewers” simply don’t feel fully qualified enough to make the necessary judgement of a “yes” or “no” vote. Sure, they are familiar with a topic to some degree, but just don’t spend their lives immersed in that corner of security.

Everyone in InfoSec “knows” about pen-testing and social engineering for example. However, unless that is their primary tradecraft and they have been doing it for a decade or more, they may not be keeping up with the latest tools and techniques. In such cases, the general reviewers will typically “defer” to the subject matter experts. The input provided by the Special Reviewers this year has been invaluable in helping shape what DEF CON 26 will be.

Discussions

The DEF CON CFP Review Board has a unique style in how they (we) review talks in contrast to many other CFP Review Boards. There is oftentimes a lot of discussion that goes on about individual talks that plays a key part in the process. The reviewers do not live in a vacuum when reviewing the individual talks, rather, they are encouraged to communicate with one another openly on the system so as to provide a higher quality of talk selection. Sometimes the discussions may turn heated, but at the end of the day it does improve the final selection. “Heated” is a really nice term. It’s a really nice term because when we say it, you may think we might mean like a “hot summer day” when it fact we mean the fires of Mordor, or whatever is causing a burning sensation in the nether regions.

The being said, on the Review Board, it’s very important to be open to new ideas and perspectives which such discussions strongly facilitate. I don’t think the DC CFP review board would work nearly as well under any other type of system. Conversely, what works for “us” may not necessarily work as well for other CFP Review Boards.

How do I get on the CFP Review Board?

First, are you really sure you want to? Do you really have the time? The numbers we posted before about the time commitment wasn’t an attempt to oversell things (in fact they are probably conservative estimates). As a review board member you will be dedicating that much time to reviewing talks over a three to five month period, with the final weeks being absolutely brutal. And if you don’t? You’ll find yourself being called out or greenlit by a shot caller. And then the best option there is you may not be asked back the following year. Remember, you are helping to shape the tone, feel, and content of DEF CON, the longest-running hacker convention now attended by over 25,000 people. That is an incredible responsibility and you are helping ensure that attendees get value from the talks they attend.

Still want to do it though? OK. Talk to some CFP Review Board members at DEF CON 26. That’s it… just do that. Judge for yourself based on how they describe it, the good and the bad. If any of them describe a breezy stroll through a nice park with flowers and chipmunks, walk away. They aren’t telling you the whole story.

Why don’t you have a CFP Review Board Panel at Defcon?

First, it would be super boring. Invariably the attendees are going to ask us a lot of questions that we can’t answer about specific submissions. While we may “vague” tweet or generally answer a question, we can’t and won’t provide specifics on submitted talks beyond what Nikita and Alex have provided as official feedback, and then only to the person that submitted the talk. So the panel would consist of a lot of jokes, high-level “CFP tips”, and not much more value. If you really want to “know” more about the CFP, just find out where some of us hangout at DEF CON.

Before we end this first entry in this series of three or four posts. I would like to take the opportunity thank you for reading along thus far. Jericho and myself worked on this entry, but he shouldn’t be held responsible for my tangents, side notes, and improper use of some punctuation.

Credit Roll

First and Foremost, we really need to thank those people around us (friends, family, significant others) that deal with us during the three to five month a year process of reviewing talks. They truly are the unsung heroes. They know we can’t go into specifics, but they’re there to listen to us bitch and moan about “that talk”. They understand us during this endeavor when we forgo plans to hangout with them or we’re not in bed until three hours past normal time. Without their support, we could never accomplish the task laid out in front of us.

General Reviewers

Jericho Roamer HighWiz Shaggy
bcrypt Vyrus Zoz Claviger
Suggy Wiseacre Secbarbie PWCrack
KingTuna Medic Dead Addict ZFasel

Special Reviewers

Andrea Matwyshyn w0nk Malware Unicorn
Snow Kodor Grifter

DEF CON Staff

Nikita Alex

DEF CON Founder

The Dark Tangent

Shoutouts

We’d also like to give a big shout out to the Workshops Review Board. While they are a separate entity from the CFP Review Board, their contributions to DEF CON are just as important.

Tottenkoph Munin Sethalump DaKahuna
CyberSulu Kodor SinderzNAshes SinderzNAshes
Kodor SinderzNAshes Wiseacre HighWiz

In part two of the series we will be covering the statistics, because that’s the type of thing that makes some of us (but especially Jericho) super wet.

With part three will come our thoughts, and comments on the Submission Form and the Questions we ask.

Part four will be some lessons we’ve learned along the way as well as ideas for improving things in the future.

One last thing, Jericho is totally the Jimmy McNulty of the CFP Review Board.


Continue reading the second blog in this series, “DEF CON 26 CFP Basic Statistics and Observations“.

A Samsung Galaxy 8, Phantom Notifications, and @Tmobile’s Dreadful Support

This is a blog of two topics. The first, a brief technical explanation of a problem with my Samsung phone after an upgrade to Android 8.0 (Oreo) pushed by T-Mobile, the subsequent debugging, and hopefully help for anyone else experiencing the issue. The second, my horrible experience with T-Mobile Twitter-based tech support.


On April 2, T-Mobile pushed an over-the-air update for my Samsung Galaxy 8 (G8) phone. In addition to a routine Android security patch level update, it also upgraded the phone to Android version 8, code-named Oreo. Shortly after the update, I started getting what I called ‘phantom notifications’, between one and six of them every hour or less. These were audible notifications that didn’t correspond with any discernible event on the phone, sometimes in quick succession. Over the course of a week, there were a few times where an icon would appear in the notification bar for a split second, making me think it was related to a specific event, but I couldn’t figure out what. I engaged with T-Mobile on Twitter, and they offered some ideas. Here is everything I did to debug and figure this out, based on their questions and my own ideas.

  • T-Mobile: SMS App Clear Data/Cache (I suspected it may be related to SMS)
  • Me: Full power cycle
  • Me: Changed default notification to determine if the phantoms are using system notification preferences (they are)
  • T-Mobile: Verify Notification Reminder functionality = OFF
  • T-Mobile: Verify no wireless/bluetooth/NFC turned on during phantoms
  • T-Mobile: Clear cache partition on phone via Debug menu
  • Verified software versions for all functionality (‘About Device’)
  • T-Mobile: Verify all apps are updated via play store
  • T-Mobile: Verify no apps from unknown sources
  • T-Mobile: Enable Developer options (did not change anything)
  • T-Mobile: Device Maintenance showed no app crashes, no hint of a problem
  • T-Mobile / Me: Phantom notifications do NOT vibrate, while SMS is configured to (so not SMS)
  • T-Mobile: No SD card in phone
  • T-Mobile: Uninstall Samsung Health (they suspected app causing this, that app isn’t on the phone)
  • T-Mobile: Backup SMS and clear all of the messages
  • Me: DND mode suppresses the phantom notifications (observation)
  • T-Mobile: Confirm I did not download ANY new apps on Sunday (day before update), Monday (day of update), or Tue – Thur (after update)
  • T-Mobile: Confirm the last time my phone worked w/o phantom notifications was Sunday and Monday before the patch (and every day prior since buying the phone)
  • Me: twice out of hundreds of times, i have seen a ‘health monitor’ type icon appear in notifications for a split second when it happens
  • Me: One-by-one disable app notifications, wait for phantom. process of elimination = found the offending app = PROBLEM SOLVED

Naturally, it was the last app on the list I had notifications enabled for. “Weather & Clock Widget for Android” by Devexpert.NET, which worked fine on Android 7.x, started causing these phantom notifications on Android 8.0. Uninstalling and re-installing did not fix it. The only reason I had allowed notifications from this app, is it would put the current temperature in the notification bar at all times. Blocking notifications for this app didn’t allow this behavior, but also stopped the phantom notifications. No factory reset needed.


Part 2; My dreadful experience with @Tmobile tech support via Twitter DM.

First, this isn’t the first time I have Tweeted and had them reach out via DM, offering support. I don’t recall having a good experience with them before, and this time certainly takes the cake on a poor experience. I am writing this up as a warning to others who might go this route, and as feedback to T-Mobile so they better understand what it is like on the customer side, and offer some tips for improving.

Perhaps the biggest problem with T-Mobile Twitter support, is their system for interacting with customers appears to be designed to resolve issues very quickly. I can’t speak to their workload, average customer engagement time, etc. But for a case like mine? I went through 22 different people over the course of seven days. On April 8, there were nine different people that cycled through to ‘help’ me. On April 7, while working with Reggie (who happened to be the only one out of 21 that I felt was truly helpful), he said he needed to AFK for 15 minutes for break, implying that someone else would take over. By that point, I knew I had already gone through seven others, so I told him I would happily wait until he returned. This high turnover rate on support staff worked against the process entirely for my case. Each time, the new person had to try to read the thread and figure out what was going on, and they rarely skimmed the thread it seemed. When I was offered a summary of my problem by the new person, it was typically wrong or left out important bits. T-Mobile needs to better identify problems that can’t be solved in ten minutes, and keep one or a few people on the case for consistency. When a customer repeatedly asks for a specific support person to re-engage, listen to them. Here is the list of people I dealt with:

  • Apr 3 – Joel Bannister
  • Apr 3 – Harley Sumida
  • Apr 3 – Ruben Hernandez
  • Apr 3 – Dee Medina
  • Apr 3 – Zach Ricketts
  • Apr 3 – Kimmi Smith
  • Apr 3 – Victor Loya
  • Apr 7 – Reggie Reese
  • Apr 7 – Harley Sumida
  • Apr 8 – Lauren Chan
  • Apr 8 – Pete Harman
  • Apr 8 – Marva Biggar
  • Apr 8 – Sora Yi
  • Apr 8 – Marva Biggar
  • Apr 8 – Kate Tomallo
  • Apr 8 – Lauren Chan
  • Apr 8 – Meghan Parks
  • Apr 8 – Eddie Gough
  • Apr 8 – Scott Degelman
  • Apr 8 – Ray Butler
  • Apr 9 – Dee Medina
  • Apr 9 – Mike Perez
  • Apr 9 – Alex Kimbrell
  • Apr 9 – Zach Ricketts
  • Apr 10 – PoxMaphixat [1]
  • Apr 10 – Kyle Saragosa
  • Apr 10 – Scott Degelman

[1] This was the only person that didn’t appear in Twitter DMs with a real name shown by Twitter:


The next bigger problem I faced, is that T-Mobile’s documentation for their support staff is out of date. It’s as if they had never debugged an issue on a Galaxy 8, despite them selling it for half a year. During the ordeal of figuring out my problem, I ran into several times where support failed related to this:

  • Apr 3 – Document for changing SMS message sounds is outdated, not correct for G8 (you apparently can’t on this model)
  • Apr 3 – T-Mobile said to set up a notification log for debugging purposes, yet G8 removed that functionality (ridiculous)
  • Apr 7 – The location of the ‘build number’ to enter developer mode is different on the G8 than previous models
  • Apr 7 – They asked me to go to the ‘Security’ screen in options, yet on the G8 that is ‘Lock Screen and Security’
  • Apr 7 – T-Mobile diagnostic data said ‘apps from unknown sources’ was enabled, my screen said it was disabled
  • Apr 8 – They asked me to check the ‘Samsung Health’ app (there is none, apparently part of the ‘Activity Zone’ app, but that function is disabled)
  • Apr 9 – T-Mobile kept telling me a factory reset is the way to fix this, despite it not necessarily working
  • Apr 10 – T-Mobile told me a factory reset is the way to go AFTER I solved the problem (WTF?!)

After having to correct the T-Mobile support staff this many times, and figure out how to find what they were looking for, it shows an obvious gap in their support ability. As someone who wrote my fair share of technical documentation, I cannot stress how important this is.

As mentioned above, when a new support person steps in, they have to skim the thread to catch up. One person told me that they take extensive notes to alleviate that problem, but after most of the new people offering me a summary got major parts wrong, I don’t think that is the case. Even if they do take notes, I think they are not consolidated, not done in a way for easy transition of the case, and generally convoluted. This causes the support staff to repeat the same things, ask the same questions, and waste customer time.

Next, T-Mobile needs to make sure their employees understand policy. Compare:

  • Apr 3 (Vinny) – “Thanks a bunch for remaining engaged with us at T-Force today, my name is Vinny and I’ll be taking over from here, as Krystn, as she had to step away.”
  • Apr 3 (Joel) – “Thank you so much for reaching out to T-Force! My name is Joel and I will be your #MagentaExpert!”
  • Apr 3 (Ruben) – “I hope you are having an amazing day. My name is Ruben and I will be taking excellent care of you and all of your concerns/questions today.”
  • Apr 3 (Zach) – “Thanks for sticking with us here. My name is Zach, and I’ll be taking over from here.”
  • Apr 7 (Reggie) – “I do want to introduce myself, my name is Reggie and I will be your #MagentaExpert today.”
  • Apr 8 (Meghan) – “My teammate had to step out for a quick meeting but my name is Meghan and I’ll be taking over to provide you with excellent service!”
  • Apr 8 (Eddie) – “Fun fact, Since T-Force is a team and constantly changes to ensure that customers always have support 24/7 we are not supposed to share our name since it already shows on the message.”

After support staff introduced themselves by name six times, Eddie came along and said they aren’t supposed to share their name. He further points out that Twitter shows their name (in the native web interface, not in Tweetdeck BTW), and yet that isn’t the case either as seen by “PoxMaphixat” above.

While some that interact with T-Mobile may say they are really ‘nice’, to me, that isn’t the case. Their overboard attempts to portray a fun and friendly atmosphere are insulting and a waste of time. Throughout the week, I was assured that they were there to help and resolve my issue, while not reading the prior messages, not understanding the issue, and bouncing in and out of my ticket to the point it was difficult keeping up with them. The phrase they loved to over-use, “I will be your #MagentaExpert!” is a joke. Seven days to figure out my problem, and they never did, I had to. Other phrases they love to say, adding fluff and not actual support, while not reading the thread and repeating the same things over and over:

  • I absolutely want to be able to help you in any way that I can!
  • It’s great seeing you here today. I hope you are having an amazing day.
  • That is an awesome question and definitely not something I am familiar with, but we can definitely work together to look into it!
  • I honestly want the best and fastest resolution for you!
  • Thank you for taking time out of your day on this!
  • Here at T-Force, we value customers time and always want to get them the best resolution possible without wasting their time.
  • We’ve got your back! (T-Mobile needs to remove this from their playbook, it is insulting.)
  • I really appreciate you reaching out and working with T-Force today.

Overall, I need a lot less of this fluffy wording, and a lot more I didn’t quote, and more actual support. If you have to keep telling me you “have my back” and want to give me the “best resolution possible”, you are convincing me you aren’t good at your job. We expect customer support to do that already.

Apr 3 (Joel) – “If you prefer to not do that, then you always have the option to back up the device and reset the software completely.”
Apr 3 (Zach) – “Can you please tell me if you’ve completed a master reset on the device since the update?”
Apr 3 (me) – “If a ‘master rest’ means a ‘factory reset’, that may be a deal breaker.”
Apr 3 (Zach) – “Typically, if there are any bugs that come across after an update, which this one may just be, a factory reset would be the best possible solution, as inconvenient as it can be to set everything up again.”
Apr 3 (Kimmi) – “In those instances the only fix I’ve been able to locate based on user feedback is a factory reset of the device.”
Apr 3 (Kimmi) – “Unfortunately the only option we have at this time is to complete the reset.”
Apr 3 (Victor) – “The master reset would be a great way to fix the issue in case it’s just some sort of temporary issue. ”
Apr 7 (Reggie) – “By no means do I want to tell you that you absolutely must do this, but in the end I want to respect your time and I feel like at this point the Master reset might fix the issue permanently whereas what we have done has demonstrably had no effect on the issue at hand.”
Apr 7 (me) – “If a factory reset is the answer, then I walk from Tmobile and go on a social media campaign to dissuade people from using Tmobile, because that is just sloppy programming and a complete breakdown of tech / customer support.”
Apr 8 (Marva) – “I know Reggie mentioned a master reset and that seems to be the only thing we haven’t tried up until this point, is that correct?”
Apr 8 (me) – “Safe mode has not been tried, and a reset, the nuclear option, is out of the question.”
Apr 8 (Sora) – ” I know that you do not want to do a master reset … I totally follow your logic; I do want to mention that if the software update is giving this error, then a master reset does allow the software to be restored on your phone properly.”
Apr 8 (Marva) – “The next step in troubleshooting is to complete that master reset.”
Apr 8 (Kate) – “The Master Reset sounds nuclear, but truly is the faster and cleanest resolution available.”
Apr 8 (me) – “As I said earlier this week, a factory reset means I will no longer be a T-Mobile customer, and will blog about this entire mess, that T-Mobile sent faulty software and could not debug it, and now is pressuring me to go that route while ignoring my direct questions about Samsung Health buginess, that icon that shows sometimes, and my desire to explore that route. That said, do you still think a factory reset is the right option instead of pursuing valid leads that may fix this without a reset?”
Apr 8 (me) – “From there, process of elimination can tell likely tell us which app is causing them. No safe mode, no factory reset. Please add this to your CS playbook.”
Apr 8 (Eddie) – “With the awesome software that we have nowadays, a master reset is the best option since there’s a high chance the bug will be deleted, and your information will be downloaded onto your phone within less than one hour if it’s backed up”
Apr 8 (me) – “Ugh, STOP. Do not recommend a factory reset to me again. I just gave a viable option to better figure this out that will take a few hours, and you go back to factory reset, after I have REPEATEDLY said that is a nuclear option and I a) will not do it OR b) do it and no longer be a tmobile customer.”
Apr 8 (Eddie) – “I just wanted to assure you that we are going to be here for you until we get a resolution. Never wanted to tell you that you should do a master reset.”
Apr 8 (me) – “I mentioned I found a new solution to this kind of problem, to add to your play book. And you immediately recommend a factory reset despite me REPEATEDLY saying ‘no’. You understand no means no right? I am tired of being told why a master reset is the option, and I am *more* tired of Tmobile reps not reading why it is NOT necessarily the right option, why it is NOT a guarantee it will fix anything.”
Apr 9 (Alex) – “If so, have you installed them and reinstalled them? Those are the first two steps, so let me know how that goes!”
Apr 9 (me) – “Two? There were *19* people on the Tmobile side during the course of this investigation, all of who gave up and told me to factory reset.”
Apr 9 (Alex) – “Now, I know we mentioned a master reset was something we should try.”
Apr 9 (me) – “Pretty much confirmed, “Weather & Clock Widget for Android” by http://Devexpert.NET is the one causing the phantom notifications. Uninstalling and re-installing it to start.”
Apr 9 (me) – “Uninstall & Reinstall did not fix it. So there is some weird issue between the app and the Oreo update. I can get around this by disabling notifications for that app, which only makes it so I don’t get the temperature in my notification bar. With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “I also explicitly said last night to STOP telling me to factory reset.”
Apr 9 (me) – “I have asked half a dozen times and every single one of you jerks ignore me. Focus on THAT problem instead of a factory reset.”
Apr 9 (me) – “With that, I have figured it out after 6 days, and without a factory reset, which half a dozen or more of your agents kept telling me to do, over and over and over…”
Apr 9 (me) – “At this point i am 99.99% sure I have this resolved, again, without a factory reset.”
Apr 10 (PoxMaphixat) – “Resetting the device and processing a warranty exchange is our last resort. Which would result in a device that is fully reset as well. This might be the thing we would need to do since we’re not able to resolve this phantom issue.”
Apr 10 (me) – “Not only have i solved the issue, I have said repeatedly NOT to recommend a factory reset to me, and you assholes keep doing it. NO MEANS NO.”
Apr 10 (Kyle) – “We can see that you’ve invested a lot of time with these issues on your phone and wanted to avoid going through the previous steps that’s you’ve already done, which is why we were looking at the master reset as a last resort … So our troubleshooting steps would basically be the master reset as well though I Samsung may have more support on what’s going on with this app.”
Apr 10 (me) – “Seriously? You suggest a master reset AGAIN when I have said over and over NOT to tell me that? I solved the phantom notification issue without a reset,”
Apr 10 (Kyle) – “I would reach out to Samsung as I completely understand your concern regarding the reset and they would be able to support the app even further. Does this make sense, Brian?”
Apr 10 (me) – “You said ‘reset’ again. How can I be any more clear here? Never, EVER, not a single time, EVER tell me to factory reset my device. Don’t even mention the word ‘reset’, let alone ‘master reset’ or ‘factory reset’. I honestly feel like there is a den of rapists and molesters working at Tmobile, who don’t understand what the word ‘NO’ means. Does this make sense, Joel / Harley / Ruben / Dee / Zach / Kimmi / Victor / Lauren / Pete / Marva / Kate / Meghan / Eddie / Scott / Ray / Mike / Alex / Zach / PoxMaphixat / Kyle?”

After this? Scott said ‘reset’ once more shortly after my last message. This is the text-book definition of the worst customer support that can be offered. A customer specifically says, over and over, not to recommend a bad support option (the factory reset). Yet, T-Mobile kept recommending it every single time. It gets to the point where it is a trigger word for me, because it clearly shows the support person didn’t read the prior messages. It means that the support staff didn’t leave a message for the next person not to bring up a factory reset. Worse? I SOLVED the technical issue, without a factory reset, and said as much. T-Mobile’s solution? Keep recommending a factory reset anyway, when it was clearly not needed. This is hands-down the worst customer service you could possibly offer, and completely insensitive to a customer. I don’t really care where the breakdown happened, other than it happened half a dozen times, but when a customer says “do not do $thing“, you should NOT do $thing. No questions, no arguments, no equivocation. Yet T-Mobile ignored that basic point, that basic understanding of the tenets of customer support. 18 separate times, reset was their answer, three times after resolving my issue.

My next advice for T-Mobile is to embrace an old classic of customer service. Over six days, interacting with 21 different support people, after repeated complaints about many of them, no manager stepped in. At least, no one identified themselves as a manager, no one exhibited any signs they were a manager, and absolutely no one made it a point to get me a resolution other than the empty “we’ve got your back” lies. Imagine going into a Taco Bell and talking with 21 employees trying to resolve a problem, that your Mexican Pizza was missing ingredients or not cooked, and that entire time no manager stepping in to ensure you got a properly prepared and cooked food item. To me, the customer, those scenarios are no different.

Finally, the bigger picture. I engaged support for one problem, the phantom notifications, which I eventually resolved myself. During the process, T-Mobile asked me questions that highlighted other problems. Despite figuring out the original, I left the engagement with two additional problems that they did not resolve. First, I asked how to disabled ‘Bixby’ completely, and they couldn’t help. Like so many other things, they didn’t understand the software, and/or their documentation wasn’t updated. I had to tell them to disable it per their instructions, it required creating a Samsung account. You actually can’t access the real settings of that malware without creating an account. That is atrocious and just bad design. Second, when we went down the road of the occasional phantom notification icon that I saw, it led us to the ‘Samsung Health’ feature within ‘Activity Zone’. On my phone, it says “tap here to get started” and tapping there does nothing. T-Mobile never helped with that, and after specifically asking them to half a dozen times, they told me to talk to Samsung.

Two more bonus observations, that came up during this ordeal. First, the T-Mobile software update downloaded over 4G, not WiFi. It used to prompt you if you wanted to wait for WiFi and this time it did not. Second, I mentioned that T-Mobile was still sending SMS notifications to me before 9 AM, and one of the support people were gung-ho saying that was not right, they would take my complaint to the top! Well, good luck there, since the last time I brought that issue up on Twitter it did go to the top, all the way to the office of the executives. Nothing ever came of it and I still get text messages from them before 9 AM. If you are going to grab that flag and head on a crusade on my behalf? Maybe consider better helping fix my original problem first.

So, T-Mobile, I have given you a wide variety of ideas for improving customer support. It is in the context of a support case you can easily reference. These ideas are very much in line with many other support services offered by similar services and companies. It’s time for you to up your game.

Ad-hoc Charity Type Things

Last month, I decided to an ad-hoc charity drive via Twitter. I did it figuring I might get a handful of donations between $5 and $25 dollars and would help out some animal charities. Boy was I shocked.

Right out of the gate, Steve Syfuhs donated $35 to the ASPCA for directly helping them previously. Almost at the same time, Steve Ragan donated $130 to Greenwood Wildlife Rehabilitation to help over 3,000 wildlife they get there a year. Wildlife rehab shops are vanishing around the U.S. as they don’t receive any state or federal funding, and rely entirely on donations and fundraising. Along with Ragan’s donation, ‘Priest’ (@imyourpriestt) donated $75 to the Georgia Society of for the Prevention of Cruelty to Animals.

With the ‘Steves’ quickly donating, I decided that at least three people would receive something from me in return for their generous donations. Before that Tweet could land, Doc Panda showed that he donated $50 to the Tiny Paws Pug Rescue, which is epic because Pugs are epic. By this point, I was working on expanding the rewards and decided that first place would get a box, not an envelope, and kept baiting for more donations. Then it hit, someone donated over $300 to a charity as part of this ad-hoc contest.

That donation to Cavy Care is amazing, because it is run by two people out of their house, with help from volunteers on weekends primarily. In the past, they have had as many as 150 guinea pigs that needed care for various reasons, usually because they are adopted as a pet for children, and they aren’t suitable for kids despite that notion. Guinea pigs are rarely adopted from animal shelters as they tend to be adults with unknown provenance or age. Cavy Care provides a sanctuary for these guinea pigs, and adopts them out very cautiously to ensure that the gpiggie finds a forever home. Cavy Care was pleasantly surprised at the sudden huge donation!

One Friday night, five amazing people, and $623 donated to charity in exchange for stickers originally. I think I sent out three boxes of stuff to the top three, and large envelopes to the other two. I cannot thank these people enough, and I hope that more will follow in these footsteps. InfoSec tends to draw large salaries. We all love our toys and our lifestyles, me included. But I think it is important that we stop a few times a year and look to help others that could benefit from our generosity.

(Disclaimer: Lebowski was not included. I would not ship that glorious beast.)

Before you publish your end-of-year vulnerability statistics…

TL;DR – The CVE dataset does not allow you to determine how many vulnerabilities were disclosed in 2017.


I’ll try to keep this fairly short and to the point, but who am I kidding? Every year for a decade or more, we see the same thing over and over: companies that do not track or aggregate vulnerabilities decide to do their own review and analysis of disclosures for the prior year. Invariably, most do it based on the publicly available CVE/NVD data, and they do it without understanding what the dataset really represents. I know, it seems simple on the surface, but the CVE dataset is not easily understood. Even if you understand the individual contents of the export, you may not understand how it was created, what shortcomings there are, what is missing, and what statistical traps you face in digesting the data. Just doing the basic parsing and automated ‘analysis’ of that data via your tool of choice (be it grep or something fancier) means very little unless you can disclaim and properly explain your results. Either way, follow along with the advice below before you publish your ‘vulnerability stats for 2017’ please!

So let’s start with the basics of CVE data analysis. Begin by grabbing the latest CVE dump, a gzipped CSV file, that represents MITRE’s CVE dataset. Note, this is different than the exports NVD offers and welcome to the first hurdle. While the base vulnerability data is 100% equivalent between the two, NVD does additional analysis and creates metadata that is useful to many organizations. NVD provides CVSS scoring and CPE data for example. The relationship between CVE and NVD is interesting if you observe it over time, where it used to be a clear ‘MITRE publishes, a day later NVD publishes’ relationship. For the last year or two, NVD will sometimes open up a CVE ID before MITRE does for various reasons. This also gave way to Bill Ladd observing and writing about how the Chinese National Vulnerability Database (CNNVD) is actually opening up CVE IDs faster than both NVD and MITRE. Consider that for a minute and understand that the relationship between these three entities is not straightforward. Then consider the relationship between many other entities in the bigger picture, and it gets even more convoluted.

See? You start by grabbing a data dump, a paragraph later you have the start of disclaimers and oddities as pertains to the larger CVE ecosystem. Next, decompress the CVE dump so you have a CSV file to work with. Now, before you eagerly start to parse this data, stop for a moment. Did you do this same analysis last year? If so, great! Do you understand what has changed in the last 18 months with regards to CVE and more specifically MITRE? If you can’t quickly and readily answer that question definitively, the kind of changes that are the first in almost 19 years for the program, reconsider if you should be commenting on this data. In case you missed it, Steve Ragan published an article about MITRE / CVE’s shortcomings in September of 2016. The article pointed out that MITRE was severely deficient in vulnerability coverage, as it has been for a decade. Unlike other articles, or my repeated blogs, Ragan’s article along with additional pressure from the industry prompted the House Energy and Commerce Committee to write a letter to MITRE asking for answers on March 30, 2017. When a certain board member brought it up on the CVE Board list, and directly told MITRE that their response should be made public, MITRE did not respond to that mail in a meaningful manner and ultimately never shared their response to Congress with the CVE Board. It is important for you to understand that MITRE operates CVE as they wish and that any notion of oversight or ‘Board’ input is only as it is convenient to them. The board has little to no real influence over many aspects of MITRE’s operation of CVE other than when they set an official vote on a given policy. Additionally, if you point out how such a vote that impacts the industry is not adopted by certain entities such as CNAs, many years down the road? They don’t want to hear about that either. It’s up to the CNAs to actually care, and fortunately some of them care very much. Oh, you know what a CNA is, and why they matter, right? Good!

OK, so you have your data dump… you better understand the state of CVE and that it is so deficient that Congress is on MITRE’s case. Now, as experienced vulnerability professionals, you know what this means! The rubber-band effect, where MITRE responds quickly and disproportionately to Congress breathing down their neck, and their response impacts the entire CVE ecosystem… and not necessarily in a good way. So welcome to the second half of 2017! Because it took roughly a year for the Congressional oversight and subsequent fallout to strongly influence MITRE. What was their response? It certainly wasn’t to use their abundant taxpayer funded money to directly improve their own processes. That isn’t how MITRE works as I far as I have seen in my career. Instead, MITRE decided to use their resources to better create / enhance what they call a “federated” CNA system.

First, spend a minute looking at the ‘federated’ term in relation to CVE, then look at the use of that term in the recently edited CNA Rules. Notice how the use of ‘federated’ in their context appears to have grown exponentially? Now check the definition of ‘federated’ [dictionary.com, The Free Dictionary, Merriam Webster]. While sufficiently vague, there is a common theme among these definitions. In so many words, “enlist others to do the work for you“. That, is quite simply, what the CNA model is. That is how the CNA model has meant to work from day one, but this has become the saving grace and the crutch of MITRE as well as the broader CVE ecosystem in the last few months. On the surface this seems like a good plan, as more organizations and even independent researchers can do their own assignments. On the downside, if they don’t follow the CNA rules, assignments can get messy and not as helpful to organizations that rely on CVE data. One thing that you may conclude is that any increase in CVE assignments this year may be due, in part, to the increase of CNAs. Of course, it may be interesting to you that at least two of these CNAs have not made a single assignment, and not disclosed any vulnerabilities in prior years either. Curious why they would be tapped to become a CNA.

OK, so you have your data dump… you know of one potential reason that there may be an increase in vulnerabilities this year over last, but you also know that it doesn’t necessarily mean there were actually more disclosures. You only know that there are more CVE IDs being assigned than prior years. Next, you have to consider the simple numbers game when it comes to vulnerability statistics. All CVE IDs are created equal, right? Of course not. MITRE has rules for abstracting when it comes to disclosures. Certain criteria will mean a single ID can cover multiple distinct vulnerabilities, and other VDBs may do it differently. It is easy to argue the merit of both approaches, so I don’t believe one is necessarily right or wrong. Instead, different abstraction rules tend to help different types of users. That said, you will typically see MITRE assign a single CVE ID to a group of vulnerabilities where a) it is the same product and b) it is the same type of vulnerability (e.g. XSS). You can see an example in CVE-2017-16881, which covers XSS vulnerabilities in six different Java files. That is how they typically abstract. Search around for a couple minutes and you will find where they break from that abstraction rule. This may be due to the requesting party filling out separate requests and MITRE not adhering to their own rules, such as CVE-2017-15568, CVE-2017-15569, CVE-2017-15570, and CVE-2017-15571. Then you have to consider that while MITRE will largely assign a single ID to multiple scripts vulnerable to one class (e.g. CSRF, SQLi, XSS), their CNAs do not always follow these rules. You can see examples of this with IBM (CVE-2017-1632, CVE-2017-1549) and Cisco (CVE-2017-12356, CVE-2017-12358) who consistently assign in such a manner. If you think these are outliers that have minimal impact on the overall statistics you generate, reconsider that. In keeping with their abstraction policy, IBM issued two advisories [#1, #2] covering a total of nine CVE IDs for unspecified XSS issues. If MITRE had assigned per their usual abstraction rules, that would have been a single ID.

OK, so you have your data dump… and now you are aware that parsing that dump means very little. MITRE doesn’t follow their own abstraction rules and their CNAs largely follow different rules. So many hundreds, likely a thousand or more of the IDs you are about to parse, don’t mean the same thing when it comes to the number of distinct vulnerabilities. That is around 10% of the total public CVE IDs issued for 2017! OK, forgetting about that for a minute, now you need to consider what the first part of a CVE ID means. CVE-2017-1234 means what exactly? You might think that 2017 is the year the vulnerability was disclosed, and the 1234 is the unique identifier for that year. Perhaps. Or does 2017 mean the year the vulnerability was found and an ID requested? The answer is yes, to both, sometimes. This is another aspect where historically, MITRE made an effort to assign based on when the vulnerability was discovered and/or disclosed to a vendor, not when it was published. Under the old guard, that was an important aspect of CVE as that standard meant more reliable statistics. Under the new guard, basically in the last two years, that standard has disappeared. Not only do they assign a 2017 for a vulnerability discovered and disclosed to a vendor in 2016 but published in 2017, but also they assign a 2017 ID for a vulnerability discovered and disclosed in 2017. Worse? They are also now assigning 2017 IDs to issues discovered and disclosed in previous years. If you need examples, here are MITRE-assigned (as opposed to CNAs that do the same sometimes) 2017 CVE IDs for vulnerabilities disclosed prior to this year; 2016, 2015, 2014, 2013, 2011, 2010, 2008, 2004, and 2002. Notice the missing years? Some of the CNAs cover those gaps! Note that there are over 200 cases like this, and that is important when you start your stats. And we won’t even get into the problem of duplicate CVE assignments that haven’t been rejected, like the first two assignments here (both are invalid assignments and that CNA should know better).

OK, so you have your data dump… you’re ready! Let loose the scripts and analysis! While you do that, I’ll save you some time and math. As of December 24, 2017, there are 18,251 CVE identifiers. 7,436 of them are in RESERVED status, and 133 are REJECTed. As mentioned above, 238 of them have a 2017 ID but were actually disclosed prior to 2017. So a quick bit of math means 18,251 – 7,436 – 133 – 238 = 10,444 entries with 2017 CVE IDs that were disclosed in 2017. This is an important number that will be a bit larger if you parse with Jan 1, 2018 data. This should be your starting point when you look to compare aggregated disclosures, as captured by CVE, to prior years. Based on all of the above, you also now have a considerable list of disclaimers that must be included and explained along with whatever statistics you generate. Because MITRE also stopped using (1) consistent (2) formatting to (3) designate (4) distinct (5) vulnerabilities in a CVE ID, you have no way to parse this data to actually count how many vulnerabilities are present. Finally, know that Risk Based Security’s VulnDB tracked 7,815 distinct vulnerabilities in 2017 that do not have CVE coverage.

Cliff notes? The CVE dataset does not allow you to determine how many vulnerabilities were disclosed in 2017. Hopefully this information helps with your article on vulnerability statistics!