A View Into DEF CON 25 CFP…

First, this post is not sanctioned by DEF CON in any way. I am a member of the CFP team who decided to keep some rudimentary statistics on the submissions this year. I did this to give the team a feel for just how many submissions we got, how many talks we accepted, and primarily to track the way we voted. This greatly assists the powers that be (the amazing Nikita) to more quickly determine which talks are well-received. Every day that I kept up on the spreadsheet, the more ideas I had on tracking. Other team members said “you should track…”, and I typically did. So this blog is to give some insight into the entire CFP process, with a solid slant on statistics about the submissions.

First, a few basics:

  • DEF CON 25 CFP opened on February 01, 2017
  • DEF CON 25 CFP closed on May 01, 2017
  • 17 talks were submitted after closing date and were considered for various reasons
  • We received 536 submissions
  • Three of the submissions were retracted by the end of CFP
  • BlackHat received 1,007 submissions this year for comparison

Next, who are we? There were technically 31 DC CFP reviewers this year, and you can read their fun profiles now (mouse over stuff here and there, call it an Easter egg)! Ten of them are considered ‘specialty reviewers’, where they typically review talks on a very specific topic such as ‘social engineering’ or ‘legal’. These are generally topics where the submissions are either too numerous and potentially murky to figure out if they are worth accepting (social engineering), or a topic that most of InfoSec aren’t really experts on, even when some of us are the #1 armchair lawyer in InfoSec. The specialty reviewers are expected to review their topic only usually, while a few are open to review multiple topics. That means there are 21 reviewers who are expected to review ‘as many talks as you can’, understanding that we may DEFER on a given submission if we feel it is out of our wheelhouse, and remembering that this is extremely time-consuming and we all have day jobs. Some of us have night jobs, and some of us have social lives (not me).

Every year we come up short on reviewers who are truly qualified to give solid feedback on a given topic. This year DC CFP put out a call for more volunteers and we hit a bit of gold, getting several new reviewers who are quality and put in a crazy amount of time. Next year? We know there are topics we need help on, so if you are sharp, kind of special(ty), or the top of your game in a popular field… come join us. I can’t stress how important this is. Instead of just working on a talk or doing a thing, you have the ability to help influence the presentations given at a conference with some 20,000+ attendees. That is a lot of power, a lot of influence, and the potential to do a lot of good. Personally, that is why I still sacrifice the incredible time I do.

Shout outs! The only way to start this paragraph is to call out Nikita for handling almost all CFP submission related emails. Incoming submissions, replies saying “you didn’t follow directions”, second-attempts, replies saying “no really you ‘brilliant hacker’, you didn’t read our guidelines”, posting them to the CFP platform, watching for the CFP team to say “I have questions” and us largely forgetting to flag it back to her, her following-up with the submitter, repeating several times in some cases, posting their replies, looking for the CFP team to ask more questions… hopefully you get the picture. The amount of work she fields in a three-month span, just related to CFP, is insane. I say that as someone who has worked more than 80 hours a week in this industry for the last twenty years. Oh, did I mention that she also voted on 60% of the talks? While five ‘full’ reviewers voted on less talks than her.

A plea! If you didn’t see the numerous Tweets and requests to get your talks in early, I cannot emphasize how much it benefits you, more than us. When a talk comes in during the first few weeks, it gives us plenty of time to not only review and ask questions, but to give feedback in the way of suggestions. In some cases, one of the team will break away from the board and work with the submitter to improve their submission. This year, I did that once with someone who’s original two submissions garnered a single yes vote. After working with them and giving feedback on how to combine the talks and hone in on the areas of interest, the re-submission received 12 yes votes and zero no votes. In an ideal world, that would happen for every submission, but a significant number of talks are submitted the last two days.

Meaningless numbers! Because our industry loves to work with statistics that they don’t fully understand or have little meaning without serious caveat and disclaimer (PPT), let me throw out a few. For the 536 submissions we received, the CFP team voted yes 1,223 times, no 3,555 times, maybe 186 times, deferred 945 times, and abstained 54 times. Again, we defer if we feel that a topic is not one we can fairly judge based on our expertise and rely on the rest of the team to review. We abstain when there is a potential conflict of interest: if we work with the submitter, we contributed to the submission, or have a negative personal past with the submitter.

Meaningful numbers! We requested feedback from the submitter 125 times and changed our votes 61 times. Working with us to answer our questions, willingness to accept our feedback, and work with us to build a better presentation benefits everyone. As Nikita tweeted, more than 60 of the accepted talks were from first-time DEF CON speakers. Given there were ~ 110 accepted talks (and 422 rejected), that is quite a lot. It is encouraging to see this many new speakers given some of the past submissions from egotistical industry veterans that felt they deserved a speaking slot on the back of a weak submission, simply because of “do you know who I am?!”

More meaningful numbers! Of the 536 submissions, 185 (34.77%) said they would release a new tool. Only 56 (10.53%) of those submissions said they would release a new exploit, and some of those claims were questionable. It is common for people submitting to DEF CON to also submit to BlackHat and/or BSidesLV. This year, 218 (40.98%) of those submissions were also submitted to BlackHat and 65 (12.22%) of them were also submitted to BSidesLV. For various reasons, often around the ability to get to Las Vegas, some submitting to BlackHat will submit to DEF CON but say that acceptable at DEF CON is contingent upon acceptance at BlackHat. This year, 36 (6.77%) talks were submitted to us with that caveat. In a somewhat arbitrary categorization, overall I felt that 200 (37.31%) of the talks were ‘red’ (offensive), 88 (16.41%) were ‘blue’ (defensive), and 38 (7.09%) were ‘black’. By ‘black’, I mean that the topic really had little merit or benefit for red-teaming and were really in the realm of criminals.

Even more meaningful numbers! Some of the most basic stats that can be generated for your ocular pleasure. First, these are arbitrary categories that were developed as we received submissions. Nothing formal and some talks were hard to classify:

From there, I broke it down further by some topics that aren’t necessarily specific to the red or blue domain. Again, kind of arbitrary and based on seeing the submissions as they came in and note that one talk may have been flagged as more than one topic:

When building a schedule over four days and across five tracks, while considering if it is better to suggest a talk for a village or alternative venue (e.g. Skytalks), Nikita has to play Tetris of sorts based on the accepted talks, the requested time, and the schedule. This is what she had to work with:

One of the more popular questions this year after an increased awareness and public discussion around diversity in InfoSec, is the gender breakdown for submissions:

Finally, a general picture of the submissions by month. Recall what it looked like for the April breakdown above and you once again get a good idea why we would like more submissions earlier in the process:

Finally, a quick note on a common perception for InfoSec conferences and talks in general. Given the drastic rise in the number of conferences popping up, there is a saturation that demands more submissions to fill the schedules. That means that veteran speakers can typically shop their talks around or be selective in where they submit based on the venue they find appealing. That also means more new speakers are submitting which results in a wide range of topic and quality of submissions. That led me to argue this Tweet and remind people that a conference can only work with what is submitted. Personally, I feel that the overall quality of submissions to DEF CON (and a couple other conferences I review for) have gone down this year and last. That means that DEF CON ended up accepting some talks that I personally did not care for.

Bottom line? If you are researching a cool topic, submit a talk on it. Have a unique perspective or done more digging on something? Share your work. Never submitted before? Submit early and let us work with you if you need it. If a security conference is lacking, it is due to the community as much as anything else.

Stalking me in Las Vegas…

dc-21-logo-sm

I fly out to Las Vegas tomorrow for the trifecta of summer security conventions held in oppressing heat. BlackHat Briefings, BSides Las Vegas, and DEF CON 21. If you want to catch up to talk about attrition.org, OSVDB, or anything vulnerability related, look for the disgruntled person likely wearing a squirrel-themed shirt. If you would like to stalk me down to catch up, chat about anything, or shank me, this friendly guide will assist you:

Wednesday

I will be at BSides in the morning to catch a few talks, mingle, and generally harass the BSides staff. Because they didn’t have enough going on putting together the entire convention. In the early afternoon I will make my way to BlackHat to register, visit a few vendor booths, and then give a presentation at 3:30 with Steve Christey in Palace 1 room. The talk is called “Buying Into the Bias: Why Vulnerability Statistics Suck”. Hopefully we will demonstrate how vulnerability statistics have sucked throughout the years, ways to improve them, and more. After the talk and any Q&A, I hope to stick around for the Pwnie awards and BarCon, before heading to either the Adobe or Tenable party.

Thursday

I will be at BSides almost all day and hope to catch a variety of talks that sound interesting. Later in the evening I will be in various places for a dinner meeting, and then may swing by the Microsoft party to harass their security people before ending up back at BSides for the after party.

Friday

I will be at Defcon and Skytalks, likely lingering around the Skytalks room if nothing else going on. At 8P, the Defcon Documentary is showing, as well as Hacker Pyramid / Hacker Jeopardy.

Saturday

I will be at Defcon and Skytalks, until 3P where I will present the Defcon Recognize Awards with Russ in Track 3. Come see who the charlatan of the year is, among other categories! That evening at 8P is the screening of the new movie “Reality Hackers” in Track 2. After that, probably doing vile things at the 303 party.

Sunday

If you haven’t found me by this time, you failed.

Building a better InfoSec conference…

There is an abundance of information security conferences out there. While the industry is drowning in these conferences, a lot of them are producing more noise than value. Increasingly, people are realizing that even a moderate security conference is a profit center. We need fewer conferences that are more topical and offer more value, whatever the price. In addition to the frequency of conferences, most of them are doing the same exact thing. There is a serious lack of creativity and forward-thinking. It was only the last few years that saw a couple conferences dedicate entire tracks to defensive security.

I have been attending security conferences for almost 20 years now. Based on my experience, as well as being on several CFP review teams, there are many aspects I want to see in the future.

  • More talks or entire tracks dedicated to sociology and human sciences, as relates to the security world. We see this from time to time, usually in passing regarding security awareness or phishing. Attacker profiling is a stronger use, but most talks are over-simplified and don’t cover new ground.
  • Talks on law and policy are more frequent lately, but they don’t seem to do much good. In the recent DEF CON 21 CFP review, we received many talks that focused on law and/or policy. There was one trend that emerged between all of them; no practical information on how the average person can truly make a difference. Sure, write your congress critters, stay informed, and all the usual advice. That hasn’t worked in the past. What else do you have?
  • Heckling should be encouraged. Several years ago, DEF CON changed to where questions or comments were not allowed during talks. The years prior, if a speaker said something that was not factual, you could quickly call them on it. It gave the audience a chance to see the error with minimal interruption. Now, questions are done after the talk, in a separate room, away from the audience. If a speaker says something inaccurate, the audience leaves thinking it was factual. This is a disservice to the attendees. Speakers must be kept honest.
  • Continuing that theme, all talks should have a mandatory 5 minute Q&A session at the least. It is rare that a speaker is so decisive and thorough as to leave no questions. If an audience member wants to debate a point or call them on bullshit, they get an opportunity to do just that.
  • More lightning talks, with a twist! Having 3 presentations in an hour gives more researchers a chance to share their progress and ideas. It gives a brief platform for them to find others that may want to help, or get ideas for moving forward. The twist? A gong. If a talk is bad or going nowhere, don’t even give them their 15 or 20 minutes. Gong them off the stage and let the next lightning talk start.
  • Most conferences solicit talks (the CFP), have a review team decide which are worthwhile, and create a schedule. It would be nice to see conferences follow this process to weed out the crap, but then put all good talks up for community vote. Based on the feedback, use it to determine what the masses want to see and then build a schedule off the higher voted talks.
  • Speakers should not only explain why they are presenting, they should justify why they are the ones giving the talk. Not a general resume with 20 years of security experience either. What specifically have they done that warrants them giving this talk. Pen-testers with a few years of experience should rarely give a talk on pen-testing or social engineering, unless they truly have groundbreaking material. They should be required to make their slides available shortly after the convention. The slides should properly reference and footnote prior work, source images, and give credit to what influenced them.
  • Conferences should solicit feedback from the audience, and give it to the speakers so that they may improve their talks in the future.

These are but a few ideas for improving conferences. Have your own ideas? Leave a comment!