A recently launched website that ranks security researchers and conferences is already questioning its future following a backlash from those working in the field.
Pwnhead.com launched at the tail end of last year, with the aim of scoring security conferences, companies and the people that work for them.
“If a conference were a movie, you can just open imdb.com and check it’s score. But there is no such thing for security conferences,” a blog post on the site explains.
The same is true of companies and their employees, the January 3 post says.
“All these problems occurs because of a one thing: There is no standardised review/scoring system in computer security scene. We built pwnhead.com as a solution for these problems,” the blog reads.
The site scores 82 global conference on a number of metrics, including its age, the number of attendees, and access to video recording. The highest ranked is DEF CON in Las Vegas, followed by Black Hat USA and Black Hat Europe. Kiwicon, hosted in Wellington, ranks 78 on the list.
Individuals were scored on their ‘technical skills’; using metrics such as the number of Github repositories and stars, the popularity of their security tools, the number of presentations they’ve given at popular conferences, number of journal papers and books published, as well as the person’s “impact on the security scene” as determined by Pwnhead’s 10 editors.
The top three security researchers, out of the 196 on the site, are Tavis Ormandy, a vulnerability researcher at Google; Dafydd Stuttard from web security testing software maker Portswigger; and red team software-maker Strategic Cyber’s co-founder Raphael Mudge.
Five ANZ-based researchers feature in the ranking: Wade Alcorn, Aldo Cortesi, Michael Skelton, Eldar Marcussen, Shubham Shah and Andrew Horton.
A company’s score is ascertained by taking an average score of the top five people that work there, the site’s explainer says. Google, Atredis Partners, Snap Inc, Spectorops and Mozilla make up the top five companies in the Pwnhead ranking.
No sooner had the site (believed to be run out of the UK) launched, than it faced criticism from the infosec community.
Some questioned the value of a ranking, mocking it as a “popularity contest”. Others were uneasy about the anonymity of the editors and opaqueness of the ranking algorithm.
Really bad idea to create a scoreboard like that, as security enthusiast I don't want to live in a world where you are monitored and/or ranked for your releases, really bad mentality— Sh0ck (@Sh0ckFR) January 8, 2019
Was this inspired by a Black Mirror episode? You're assigning a "social score" to researchers... The academic world tried this with several type of metrics and it's a bad idea.— Mathy Vanhoef (@vanhoefm) January 12, 2019
I do not like this idea or the sketchy execution. This is not transparent, not impartial, and not very useful. I would like to proactively opt out forever. This really isn't representative of the culture I enjoy in infosec.— Kody (@KodyKinzie) January 13, 2019
Yeah I'm about to go off here. This is biased as fk and it GROSSLY misrepresents the community and how much we value NEW ideas. More "gatekeeping" through bias. #WeakAF— SHADOWOPS | 🍥 (@Shadow0pz) January 13, 2019
A number of those ranked on the site requested to be removed from the list.
The site’s Twitter account initially responded that “it’s just fun” adding that it had made its metrics very clear and would soon be making public the formula behind the rankings.
Commonwealth Bank of Australia’s senior manager for penetration testing, David Jorm, took to Twitter to say he would not be looking kindly on Pwnhead listed researchers.
“I've been hiring pen testers and security engineers for seven years,” he said. “Everyone on the Pwnhead list is now on my auto-delete list when applying for roles. Please ask them to remove you then re-apply.”
Today, just over two weeks since it launched, the site admitted its shortcomings on Twitter.
“Our intention was creating a good value for security community. But it seems we’ve failed to do it,” the tweet read.
A tweet poll – asking users whether Pwnhead should continue unchanged or ditch the people ranking element of the site – has currently generated more than 400 votes, 80 per cent of which want the individual rankings removed.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.