Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
Comment by sab —
November 21, 2008 @ 11:50 pm
Thanks for the post. Where do these folks in NY think dead turkeys for holidays come from?
Comment by Jim Henley —
November 22, 2008 @ 12:02 am
The store!
Comment by ari —
November 22, 2008 @ 12:36 am
My favorite part of the video is the occasional gobbling and the look on the guy’s face, which seems to be saying: “Can I behead this bird now? Or is that going to cause me trouble with the governor?”
Comment by buermann —
November 22, 2008 @ 5:11 am
I was impressed, myself. When I was growing up we just had a trunk and a hatchet and feathers, everywhere.
Comment by josephdietrich —
November 22, 2008 @ 11:28 am
Why shouldn’t he be the backdrop of an interview?
Oh, I don’t know, maybe because it’s incredibly distracting? But that’s just my impression.
Comment by geeno —
November 22, 2008 @ 12:09 pm
It’s just the raw funny that makes this. She pardons one turkey in a kid-friendly gesture, then stands there giving this stupid interview while the guy kills the rest of them. It’s Palin’s almost surreal cluelessness about how things look that is just flat out hilarious.
I agree that anyone feigning offense at this is just looking for something to bother themselves about, but c’mon – it’s funny.
BTW – I’m from NY and we get wild turkeys around here all the time. It’s not hard to do the math from there.
Comment by geeno —
November 22, 2008 @ 12:14 pm
Commenter at John Cole’s
“You wouldn’t do a Mother’s Day interview in front of two people f*cking and defend it by saying ‘Where do you think babies come from?’”
Comment by Avram —
November 22, 2008 @ 5:00 pm
My favorite part is the reporter asking if there are any programs “on the chopping block”.
Comment by kid bitzer —
November 22, 2008 @ 6:18 pm
wait, how come i have to be a vegan in order to have a quarrel?
why a vegan?
(why a duck?)
can’t ovo-lacto vegetarians decry the killing of animals for food?
(and geeno–that comment from john cole’s thread is killer funny!)
Comment by Thers —
November 23, 2008 @ 2:37 am
Well, I’m in NY, and I can tell you exactly where the chicken in our basement freezer comes from. Our backyard.
Yeah, I don’t know who’s “offended” by the video, as opposed to people thinking “this is not really very astute political theater.”
If Palin had thought this through and were making an argument about factory farming etc. in the video that would be one thing, but I kinda don’t think she had that in mind.
The guy should be the *subject* of an interview. I’d be interested. Palin, not so much.
Comment by excathedra88 —
November 23, 2008 @ 3:06 am
Most folks seem to be missing the point, which is not the ludicrous degree of hypocrisy demonstrated by those that sup on fowl decrying the brutality of the great turkey murder video, nor the apparent political naivety of the Palindrome.
Nope, the point is she plays to her base, which ain’t liberal bloggers, vegans ( or lacto-ovos), progressives, or even creative anarchists. Good old ‘red blooded’ ‘Mericans! ( notably, older hetro males, and the women who subordinate themselves to such).
Remember F. Thompson’s field dress a moose comment? The ones applauding and laughing at that comment – them be her base, and the entire episode at the turkey farm plays right to them ( personally, I prefer a pol who knows how to field dress a Washington lobbyist, and eagerly await such a video).
Comment by Chuck Butcher —
November 23, 2008 @ 6:08 am
The issuing of plutocrat tags would make for interesting field dressing and probably very long lines.
Comment by Steve —
November 23, 2008 @ 7:33 am
She was just introducing America to Joe the Turkey Processor. He will be on stage with her during the next election.
Comment by Jim Henley —
November 23, 2008 @ 10:44 am
@kid bitzer: For serious, it depends. Specifically on where the eggs and milk come from. Laying hens are probably the most abused animal in the industrial system – pigs give them a run for the title, I’ll grant you – and dairy cows can fair nearly as poorly.
Cruelty-free eggs are readily available, though. Milk is a tougher issue.
The above presupposes the Jim value system that subjecting an animal to life-long torture is worse than simply killing it.
Comment by Libby Spencer —
November 23, 2008 @ 10:46 am
Having spent 15 years on the farm and seen chickens beheaded with an axe, I wasn’t offended. Actually I was pretty impressed with the cone of decaptitation. Much less messy.
I was somewhat offended by her shameless display of naked ambition though. If it’s customary for governors to pardon turkeys, I’m unaware of it. Rather arrogant co-opting of a presidential tradition, I thought.
Comment by Jim Henley —
November 23, 2008 @ 10:53 am
Yeah, cones are the standard now for the small producer. The farmer who raises the bulk of the Henley family chickens has them. Not sure who brought the innovation when.
Comment by Jen —
November 23, 2008 @ 12:37 pm
While reading the proclamation inside the turkey enclosure:
“I, as governor, a friend to all creatures big and small…”
“in Alaska, where we don’t even have the death penalty”
“I was happy to get to be invited to participate in this. For one, you need a little bit of levity in this job, especially with so much that has gone in the last couple of months that has been so political obviously that it’s nice to get out and do something to promote a local business and to just participate in something that isn’t so heavy-handed politics that it invites criticism. Certainly we’ll probably invite criticism for even doing this, too, but at least this was fun.”
And also because then ya know too I also enjoy drinkin’ my latte also while doin’ the interview then watchin’ the turkeys get their blood drained. You betcha!
Comment by tjproudamerican —
November 23, 2008 @ 2:48 pm
For those who think this video is not a disaster, do you anticipate other politicians being interviewed while animals are slaughtered in the background?
Mark Steyn says this video is so impressive that it is Palin’s first ad in her 2012 race. Are we, thanks to he men Conservatives like Sarah, in for an honesty in advertising run? Will we see beef ads showing CEO’s on the Slaughterhouse Floors?
Comment by tjproudamerican —
November 23, 2008 @ 2:55 pm
One more point:
I never joined the chorus who denounced Michael Vick. I think “animal cruelty” laws are a joke. It isn’t just “liberals” who get weepy-eyed over animals. Conservatives were also denouncing Michael Vick.
As a meat eater, I say who cares how the animals die. As long as they taste good.
But Palin looks stupid with someone upstaging her and Americans don’t want to know how their food is processed.
Comment by Kurzleg —
November 23, 2008 @ 7:37 pm
TJPA –
On the contrary. I think a growing number of people want to know how their food is processed, and I think food processors want to keep that hidden from view as much as possible for obvious reasons. It is a problem when people are kept uninformed about how their food is raised because it prevents them from making better choices about what food they buy.
As for your comment about not caring how animals die, I truly feel sorry for you. If you can’t empathize with the fate of a fellow living creature, then you’re truly an emotionally impoverished human being.
Comment by joe from Lowell —
November 23, 2008 @ 8:20 pm
I agree, this seems to be a decent, reasonably humane type of meat-raising, much better than the nightmarish factor farms. Uh, so?
Why doesn’t she just do the next interview while taking a shit?
It’s just an ordinary bodily function. There’s nothing wrong with it at all. Where do people from New York – who, apparently, are extremely stupid – think the food they eat goes?
What are you, kidding me? What’s wrong with giving a press interview while livestock is being slaughtered in the background? What are you, kidding me?
Comment by dhex —
November 24, 2008 @ 10:32 am
eh, i still don’t think it’s that big a deal.
and i’m from new york, where turkeys come from whole foods at 11 bucks a pound.
Comment by Jim Henley —
November 24, 2008 @ 11:11 am
Wow. I thought the $4/pound 22-pounder I got from Mrs. Bolton on Saturday was an expensive bird!
Comment by Mike —
November 24, 2008 @ 12:33 pm
It’s pretty obvious what she’s up to here – it’s simply theater, put out there hoping someone makes a big deal about it so the base can be treated to comments about the sissy liberals from NY who don’t have a clue where their food comes from. The base can laugh at the weeny liberals. Yawn.
She’s busy running for president in 2012. Hopefully, her 15 minutes will end long before that.