Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.

The death toll from Britain’s biggest freeze for decades reached 22 today as the country prepared for its coldest night so far, bringing the promise of even more treacherous conditions.
Thousands of homes have been left without power, schools have closed and travellers have faced chaos as the weather hit roads, rail services and airports over the last two days. The disruption is estimated to have cost businesses around £700 million.
Councils continued to struggle with a growing salt emergency as police warned drivers in many areas not to travel unless their journey was essential.
The AA expect to have attended 20,000 breakdowns today - compared with about 9,000 for a normal Thursday - and warned that conditions were expected to remain “treacherous”.
Meanwhile, the shutdown of an offshore Norwegian gasfield pushed Britain's gas infrastructure into emergency mode, forcing the closure of industrial companies in the north of England in order to preserve supplies to homes, shops and offices.
Although major airports stayed open, some air passengers had long waits for their flights, particularly at Gatwick, on the outskirts of south London, where more than 130 flights were cancelled. EasyJet had to axe more than 100 flights and British Airways was among other carriers that had to cancel some services.
The body of Philip Hughes, 45, from Slough, was recovered from beneath ice at the Lakeside Country Club in Surrey where he was watching the the world darts championship. A spokesman said it appeared to have been a “tragic accident”.
His death brings to 22 the toll of people killed by conditions related to the weather since the cold snap began on December 18.
One of the suppliers of rock salt, Cleveland Potash, said it was struggling to meet demand with a spokesman saying that Cheshire’s salt mine, the biggest supplier of rock salt in the country, only had a few days’ supply left.
Harrow council, in north-west London, described its wait for supplies of rock salt as “pretty outrageous” as it came close to completely running out.
In the House of Commons, Sadiq Khan, the Transport Minister, said the Government was doing “everything possible” to keep the UK’s road network open.
However Boris Johnson, the Mayor of London, said: “This is not a dress rehearsal, it is the real thing, and everything must be done to get the supply moving and avoid the situation of London running out of salt to keep the roads open.”
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY
