Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

Posted on June 28, 2011, 6:00AM | Charles Oliver
A court in Taiwan has sentenced a blogger to 30 days in jail and two years probation for writing a negative restaurant review. The blogger, whom media identified only by her surname Liu, wrote that the restaurant's noodles were too salty and that it had cockroaches. The court said she should not have said the restaurant's food was salty because she tried only one dish during just one visit. The court also ordered her to pay NT$200,000 to the restaurant in compensation for lost business.
Reason needs your support. Please donate today!
Posted on June 27, 2011, 6:00AM | Charles Oliver
A jury convicted former Washington County, Missouri, sheriff's department chief deputy Vernon Wilson of violating the civil rights of four inmates. Wilson beat two of the inmates and slammed their heads against concrete walls. He paid inmates in cigarettes to beat the other two.
Reason needs your support. Please donate today!
Posted on June 24, 2011, 6:00AM | Charles Oliver
A federal jury has awarded Tricia Wachsmuth $30,001 after finding Powell, Wyoming, police officers forced her to walk in front of them as a human shield while searching her home and unsafely used a flashbang grenade during the search.
Reason needs your support. Please donate today!
Posted on June 23, 2011, 6:00AM | Charles Oliver
Officials at Donald A. Wilson Secondary School in Whitby, Canada, suspended Jack Christie for 20 days because of satirical animated videos he posted on YouTube. The videos, which were also investigated by local police, include a mock political candidate promising to invade Sweden and seize its attractive women. Christie says that since media picked up the story of his suspension the number of hits on his videos has quadrupled.
Reason needs your support. Please donate today!
Posted on June 22, 2011, 6:00AM | Charles Oliver
The Oregon House of Representatives has voted to ban the sale of "any substance or object that is capable of causing death to another person" for the purpose assisting someone to commit suicide. They seek to stop the sale of suicide kits, consisting of a plastic hood and tubing, that at least one Oregon resident has used to take his life.
Reason needs your support. Please donate today!
Posted on June 21, 2011, 6:00AM
San Francisco's Animal Control and Welfare Commission has proposed that the city ban the sale of pets, including goldfish. The move is aimed at reducing impulse buys of animals as well as fighting what commissioners see as inhumane breeding practices.
Reason needs your support. Please donate today!
Posted on June 20, 2011, 6:00AM | Charles Oliver
Nogales, Arizona, police officer Pedro Molera has sued his department and former sergeant Sergio Bon after Bon Tasered Molera's penis.
Reason needs your support. Please donate today!
Posted on June 17, 2011, 6:00AM | Charles Oliver
When teachers at England's Nathaniel Newton Infant School spotted two 7-year-old boys playing army, they immediately broke up the game, then they contacted the boys' parents and let them know such threatening behavior is "unacceptable."
Reason needs your support. Please donate today!
Posted on June 16, 2011, 6:00AM | Charles Oliver
A Broward County, Florida, sheriff's deputy spotted Robin Brown when she was bird watching one day. He thought that the sage she had with her was marijuana, and a field test seemed to confirm that. He didn't arrest her then, but confiscated the sage and sent it to the crime lab for testing. Unfortunately for Brown, Assistant State Attorney Mark Brown didn't wait for the sage to be tested before getting a warrant issued for her arrest. A deputy showed up at her place of work and took her away in handcuffs to jail, where she was subjected to a body-cavity search. It was a month later before her attorney found the sage had never been tested. The test confirmed it was not marijuana, and the state dropped all charges.
Reason needs your support. Please donate today!
Posted on June 15, 2011, 6:00AM | Charles Oliver
Over the past five years, police officer Daniel Alvarado has been suspended four times and reprimanded or cautioned at least 12 times, including six times for not obeying orders. So his supervisors in Texas's Northside Independent School District Police Department should not have been surprised when he disobeyed an order to stay with a teenage boy he'd just seen get punched at a bus stop. Instead, Alvarado chased the boy he had seen throw the punch, 14-year-old Derek Lopez. Minutes later, Alvarado shot Lopez, who was unarmed.
Reason needs your support. Please donate today!
Posted on June 14, 2011, 6:00AM | Charles Oliver
In Massachusetts, Westfield High School officials suspended Tom Costello and Ryan Angco and barred them from attending their graduation after the duo performed a mock battle with a Star Wars theme in the school cafeteria. Principal Raymond Broderick said the two could well have hit someone with their toy light sabers.
Reason needs your support. Please donate today!
Posted on June 13, 2011, 6:00AM | Charles Oliver
After finding the Christian owners of a bed and breakfast guilty of discrimination for turning away homosexual guests, Britain's Equality and Human Rights Commission (EHRC) says it will decide whether "gay only" guesthouses also violate the law. The EHRC says it may take action against hospitality businesses that discriminate against heterosexuals.
Reason needs your support. Please donate today!
Posted on June 10, 2011, 6:00AM | Charles Oliver
Several people saw four men push 14-year-old Dafarus Coleman and take his jacket aboard a Metro bus in King County, Washington. But when some of them called 911 to report a crime in progress, dispatchers refused to send a deputy. "You cannot report a theft that did not occur to you. The person who the items were stolen from has to report this," a dispatcher told one caller.
Reason needs your support. Please donate today!
Posted on June 9, 2011, 6:00AM | Charles Oliver
The French government has banned TV broadcasters from mentioning Facebook or Twitter unless it is part of a story. That means TV stations can't direct viewers to their Facebook page or Twitter feed.
Reason needs your support. Please donate today!
Posted on June 8, 2011, 6:00AM | Charles Oliver
Officials in Glasgow, Scotland, sent a letter to parents of students at King's Park Secondary School warning them not to allow their children to wear short skirts or tight pants because that could arouse the interests of pedophiles. The letter came after a King's Park man pleaded guilty to posing as 14-year-old girl to entice children to perform sex acts in front of their web cameras.
Reason needs your support. Please donate today!
Posted on June 7, 2011, 6:00AM
Officials in Charlotte, North Carolina, fined the Albermarle Road Presbyterian Church $4,000 for improperly pruning trees on the church's property. Tom Johnson, the city's senior urban forester, said the church can escape the fine by replacing the trees.
Reason needs your support. Please donate today!
Posted on June 6, 2011, 6:00AM | Charles Oliver
Taiwanese prosecutors have vowed to crack down on those caught hoarding food. The move is part of the government's attempts to stabilize rising food prices.
Reason needs your support. Please donate today!
Posted on June 3, 2011, 6:00AM | Charles Oliver
Police have warned residents of Kent and Surrey, England, not to put mesh wire on the windows of their sheds. They say the wire could injure burglars who try to break into the sheds.
Reason needs your support. Please donate today!
Posted on June 2, 2011, 6:00AM | Charles Oliver
Bloomington, Indiana, police officer Scott Oglesby has left the department after he picked up a 7-year-old boy by the throat. Oglesby was at a local elementary school on an unrelated matter when the special needs student began to scream and throw a tantrum. He said the boy was giving him a headache and grabbed him.
Reason needs your support. Please donate today!
Posted on June 1, 2011, 6:00AM | Charles Oliver
The Maricopa County Sheriff's Office did not adequately investigate more than 400 sexual crimes, according to a report released by the sheriff's office. Senior officials were aware of the problem for at least two years, but the report says Chief Deputy David Henderschott blocked an internal investigation.
Reason needs your support. Please donate today!
Posted on May 31, 2011, 6:00AM | Charles Oliver
Maryland has banned shipping tobacco and tobacco products directly to customers. That's the result of a law passed last year requiring tobacco merchants to have a state license. The law is supposed to make sure that customers pay the state tax on tobacco, but it bans shipments even to those who set up accounts with dealers and pay the tax.
Reason needs your support. Please donate today!
Posted on May 30, 2011, 6:00AM | Charles Oliver
Toronto, Canada, Constable Christopher Hominuk has pleaded guilty to threatening bodily harm. Hominuk was caught on video threatening to Taser the genitals of two suspects and slapping one of them in an effort to get them to help him catch a third suspect in a break-in.
Reason needs your support. Please donate today!
Posted on May 27, 2011, 6:00AM | Charles Oliver
A British immigration officer, who wasn't named by media, has been fired for putting his wife on the nation's terrorism watch list. The man added her name to the list when she was in Pakistan visiting family. When she tried to return home, she was barred from getting on the plane. The man's actions were not discovered until three years later, when he was being vetted for a promotion.
Reason needs your support. Please donate today!
Posted on May 26, 2011, 6:00AM | Charles Oliver
A woman has sued police in Christchurch, New Zealand, for using bolt cutters to remove five piercings from her body, including one in her genital area. The woman, who wasn't identified by local media, had been charged with obstruction but was later acquitted of that charge.
Reason needs your support. Please donate today!
Posted on May 25, 2011, 6:00AM | Charles Oliver
James Tate and some friends snuck onto the campus of Connecticut's Shelton High School and taped a message on the entrance asking classmate Sonali Rodrigues to go to the prom with him. Rodrigues was charmed by the gesture and agreed to go with him. School officials weren't charmed. They suspended the students and banned them from the prom. After media picked up the story, however, they reversed that decision.
Reason needs your support. Please donate today!

Site comments/questions:
Media Inquiries and Reprint Permissions:
(310) 367-6109
Editorial & Production Offices:
3415 S. Sepulveda Blvd.
Suite 400
Los Angeles, CA 90034
(310) 391-2245