Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

Airport tyranny comes to your door, as predicted

Paul Joseph Watson
Prison Planet.com
Friday, January 8, 2010
Naked body scanners are being readied to go mobile and scan you on the street, at football games and any other event where masses of people are congregated, according to a leaked paper written by Dutch authorities.
As we have been warning all along, the tyranny now being metered out at airports was always intended to be rolled out onto the streets, with mobile metal detectors already being stationed at various transport hubs in the UK in the name of stopping knife crime.
Now Dutch police have announced that they are developing a mobile scanner that will “see through people’s clothing and look for concealed weapons”.
According to a confidential document, “The scanner could first be used as an alternative to random body searches in high risk areas. The mobile detector would enable the search to be carried out more quickly and would only be used on people suspected of carrying concealed weapons,” reports Dutch News.nl.
The device would also be used from a distance on groups of people “and mass scans on crowds at events such as football matches.”
“The biggest challenge is making it portable and ensuring it can carry out a scan in seconds,” Giampiero Gerini, a professor at Eindhoven University, told the paper.
The aim is to develop and deploy the device within three years. With police in major American and British cities already carrying out random searches of innocent people under routinely abused terrorism laws, mobile scanners are likely to be added to their arsenal, especially if people have been trained to accept their use as routine in airports.
(ARTICLE CONTINUES BELOW)
Three years ago, leaked documents out of the Home Office revealed that authorities in the UK were working on proposals to fit lamp posts with CCTV cameras that would X-ray scan passers-by and “undress them” in order to “trap terror suspects”.
“The questions are when is this a useful addition to security and when does it become unduly intrusive and worrying to the public?” said Professor Paul Wilkinson, a terrorism expert.
Since everything that we see being installed at the airports is now gradually being introduced on the streets, how long will it be before mind-reading devices that scan individuals for behavioral psychology, now being discussed for use in airports, are stationed on every major street corner?
The technologies now being prepared not just for the airport, but for our everyday lives, are far more frightening and technologically advanced than anything George Orwell wrote about in 1984. Unless we stand up in unison and say enough is enough, our world will become a literal hi-tech prison grid characterized by a caste system of slaves and controllers.
Comments are closed.
PRISON PLANET.com. Copyright © Free Speech Systems LLC 2011. All rights reserved. Legal Notice. Digital Millennium Copyright Act Notice.
Home » Featured Stories » Now Mobile Devices Will Scan Your Naked Body On The Streets

