Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
Didn’t read it
I think that’s why I blog for the air. Maybe there’s at least one person interested in what I write.
I mean: I’m in the opposite side, my blog is not famous and I don’t know so much about the few visits there are… I blog just because I think I’m contributing to someone
Its true.. Its the state you have reached that makes you addicted to measuring
btw very witty title.. it made me anxious
Life is not something to be measured. Life is something to be enjoyed.
I agree with that statement to a degree. I often check things and comments on my blog then I do actually “doing” anything productive.
great article. It’s funny because even those of us who aren’t in ‘publishing’ or professionally putting out stuff also have this thing where, I make a slightly joking status, and then I check the facebook likes on it, the twitter faves on it, regular communication can become measured in the strange professional way that TV shows are. The trick is just to make the conscious decision to be yourself organically rather than become a slave to going for what’s going to get the views, the feedback, the immediate results. One thing I’ve been thinking about in that front is just separating out your accounts–this is your personal twitter account in which you say whatever you want regardless of whether the audience is ‘interested’ in each topic; that’s your professional account in which you deliver stuff that’s actually tailored to the followers, etc.
That’s so true – when my book was first published, I checked the Amazon ratings several times an hour!
I just had to stop as it was driving me crazy. There are some things that are just best left alone – I think that there are times when you need to see if what you are doing is making an impact, but it should only be a tiny proportion of the time you actually spend trying to make that impact.
I guess that finding the right, healthy balance is key.
Well Matt
I used to do this to until I noticed that I was once of the best in my niche (military history) – now I don’t even give a look anymore to the stats because I had even to use a pay for access (1$/month) to reduce the traffic in my blog. I thought this would work but after 1 month they all do pay the buxe to get access … so I would say – forget the stats and work – that’s the only way things goes on here
Gunter
Belgium
Hi Matt-
LOVE the look and feel of your blog. What stat plug-in are you using. I miss the free stats that came with wordpress.com. Since I’ve switched to self-hosted, I can’t seem to find/correctly install and use something that is as easy and works the same way. Since you are into your stats, I thought you might be able to help.
Thanks!
Install Jetpack and you’ll be set.
Well, I guess there is no Google analytics for your personal goals in life. Not yet, rumors are going on that they are working on it….
Oh nice! I was thinking how come you end up being so rude .
Pingback: I don’t care if you read this article — Goannatree
” And I was reminded again that the proudest achievements in life just can’t be measured.” The thing is that everything in life is measurable but some of these are understood and make sense because of the context and the greater pattern recognition abilities we humans improved over thousands of years.