A was asked if I could share the files of my last diary entry: Domain Whitelisting With Alexa and Umbrella Lists.
You can find the files on my site here. And to teach you how to fish :-), here are the commands I used to produce these lists:
csv-cut.py -s \t 1 emd.txt blacklist.txt
csv-lookup.py -s , -e blacklist.txt 0 top-1m-umbrella.csv 1 0 blacklist-umbrella.csv
csv-lookup.py -s , -e blacklist.txt 0 top-1m-alexa.csv 1 0 blacklist-alexa.csv
My csv tools can be found on my Beta GitHub repository.
My assumption when I read this blog post, was that the blacklisted domains would rank low in the Alexa and Umbrella lists. They dont, look at the histograms of the rankings.
Blacklisted domains with Alexa rank:
Image may be NSFW.
Clik here to view.
Blacklisted domains with Umbrella rank:
Image may be NSFW.
Clik here to view.
These long tail distributions indicate that blacklisted domains with higher ranks are more prevalent than those with lower ranks. This is also reflected in the ranking average: 350553 for Alexa and 420846 for Umbrella.
Conclusion: dont use Alexa and Umbrella top 1,000,000 lists as whitelists blindly, even if you just use the top 1000 or 10000.
Didier Stevens
Microsoft MVP Consumer Security
blog.DidierStevens.com DidierStevensLabs.com
