| Jul | AUG | Sep |
| 14 | ||
| 2010 | 2011 | 2012 |
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What?s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.

The rightwingers, including Binyamin Netanyahu, the former Prime Minister, are commemorating the bombing of the King David Hotel in Jerusalem, the headquarters of British rule, that killed 92 people and helped to drive the British from Palestine.
They have erected a plaque outside the restored building, and are holding a two-day seminar with speeches and a tour of the hotel by one of the Jewish resistance fighters involved in the attack.
Simon McDonald, the British Ambassador in Tel Aviv, and John Jenkins, the Consul-General in Jerusalem, have written to the municipality, stating: “We do not think that it is right for an act of terrorism, which led to the loss of many lives, to be commemorated.”
In particular they demanded the removal of the plaque that pays tribute to the Irgun, the Jewish resistance branch headed by Menachem Begin, the future Prime Minister, which carried out the attack on July 22, 1946.
The plaque presents as fact the Irgun’s claim that people died because the British ignored warning calls. “For reasons known only to the British, the hotel was not evacuated,” it states.
Mr McDonald and Dr Jenkins denied that the British had been warned, adding that even if they had “this does not absolve those who planted the bomb from responsibility for the deaths”. On Monday city officials agreed to remove the language deemed offensive from the blue sign hanging on the hotel’s gates, though that had not been done shortly before it was unveiled last night.
The controversy over the plaque and the two-day celebration of the bombing, sponsored by Irgun veterans and the right-wing Menachem Begin Heritage Centre, goes to the heart of the debate over the use of political violence in the Middle East. Yesterday Mr Netanyahu argued in a speech celebrating the attack that the Irgun were governed by morals, unlike fighters from groups such as Hamas.
“It’s very important to make the distinction between terror groups and freedom fighters, and between terror action and legitimate military action,” he said. “Imagine that Hamas or Hezbollah would call the military headquarters in Tel Aviv and say, ‘We have placed a bomb and we are asking you to evacuate the area’.”
But the view of the attack was very different in 1946 when The Times branded the Irgun “terrorists in disguise”. Decades later, Irgun veterans are unrepentant. Sarah Agassi, 80, remembers spying in the King David Hotel.
She and a fellow agent posed as a couple. They danced tangos and waltzes, sipped whisky and wine while they cased out the hotel.
On the day her brother and his fellow fighters posed as Arabs delivering milk and brought seven milk churns, each containing 50kg of explosives, into the building. Ms Agassi waited across the street until her brother rushed out. She said that she then made the warning call to the British command in the hotel.
Sitting in the luxurious hotel lobby, she expressed no regret. “We fought for our independence. We thought it was the right way . . . If I had to fight for Israel, I swear even now I would do anything.”
TWO VERSIONS
The original wording:
The Hotel housed the Mandate Secretariat as well as the Army Headquarters. On July 1946 (sic) Irgun fighters at the order of the Hebrew Resistance Movement planted explosives in the basement. Warning phone calls had been made urging the hotel’s occupants to leave immediately. For reasons known only to the British the hotel was not evacuated and after 25 minutes the bombs exploded, and to the Irgun’s regret and dismay 91 persons were killed.
The amended version
. . .Warning phone calls had been made to the hotel, the Palestine Post and the French Consulate, urging the hotel’s occupants to leave immediately.
The hotel was not evacuated, and after 25 minutes the bombs exploded. The entire western wing was destroyed, and to the Irgun’s regret 92 persons were killed.
Contact us | Terms and Conditions | Privacy Policy | Site Map | FAQ | Syndication | Advertising
© Times Newspapers Ltd 2010 Registered in England No. 894646 Registered office: 3 Thomas More Square, London, E98 1XY
