Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
What does one do when it has been over the 90's for over a week - you pretend to life isn't worth living anymore (and that's because Mom won't take me to my park in this heat!) So instead you flop on floor in a lifeless, motionless state hoping Mom will move away with that flashy thing and just leave a boy in peace to suffer silently. Sigh - and it isn't even summer yet. How do I convince Mom that doing my tricks and learning even more new ones just isn't the same as a walk in the park and getting all those wonderful smells. Even a trip in the air-conditioned car to the air-conditioned pet store didn't do anything. I still came home all dejected and flopped down in my air-conditioned house. Why can't they make air-conditioned parks - that is what I want to know?
19 comments:
We have been doing a lot of snoozing in the a/c. Mom takes us for shorter walks, but it just isn't the same. Bring back the spring or for us even the winter would be wonderful, but Mom says too early for winter.
Hope it cools down soon.
Woos ~ Phantom, Thunder, and Ciara
man alive, you guys are really having a heat wave out there arent you.
I am sending some cool Rocky Mountain Air your way. I hope it reaches you.
I don't think I could stand those kind of temps...
I think I would move in and live under the swamp cooler.
Be cool
Bert
So sorry it is too hot to go to the park! It is going to be one long summer! WOOF!
This looks eerily similar to Havi's pig on a stick formation...
www.twogradstudentsandapittie.blogspot.com
Air-conditioned parks? I'd vote for that!!!
Until then, at least there's AC indoors. :)
Poor hot dog, Reilly :(
Get on a bus and come see us. Cold and rainy here!
Wyatt and Stanzie
We're getting a taste of the 90's today, Reilly, and we're not looking forward to it AT ALL! We hope your heat spell breaks soon so you can head to your doggie park!
Love ya lots
Maggie and Mitch
I think you're onto a million $ idea there Reilly!
HOpe you get a cold front soon.
Aw! Keep cool! There's a looooong summer ahead of us!
Same problem here. I don't even get car rides anymore because momma says, "Oh Mango, I know that this trip into the library should only take a minute, but what if something happens and you are in the car without the AC too long?" So, yeah, lots of short walks and flopping.
Slobbers,
Mango
We try to walk at the park early. Otherwise its just to hot. I hope it cools down soon so you can walk again.
Air-conditioned parks - what a brilliant idea Reilly!!! It is crazy how it was too cold for so long and now it's already too hot!
Yea Reilly...why don't they? Katie didn't get to go to the park last night either, because it is too hot for her to be out there. She's depressed. But she has Aunt Vi to follow around now, so at least she has a good job.
We are entering a heat wave after a couple of nice days. Sending some cool breezes your way.
Katy & Bailey
It's always hot here so I'm quite used to it and that's why I don't have much long walks either. I think you need lots of ACs at home!
~ Eva
Our boys are suffering the same way - there isn't much left to do! Mom did move the agility equipment into the pool room - but that only lasts so long too!
Sam
Is adorable!!!Jajajaja
A walk in the park is the very bestest! We went for a walk in the park but the Mum said she was afraid of us falling in the river and drowning so we had to mostly stay on leash anyway. :(
Too hot ain't no fun but it's more bearable being a smoothie than a rough. But you can't be a smooth sheltie!
Post a Comment