The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
TIMESTAMPS
The Wayback Machine - https://web.archive.org/web/20110811033440/http://thecookshack.blogspot.com/2009_12_13_archive.html
The Cook Shack--Gab & Grub
JUST A GOOD OLD FASHIONED PATRIOTIC BLOG, with
Backwoods straite tawk, views, wizdom & recipees frum the man that duz the cookin...."Cookie". Scroll Through "SEARCH THIS BLOG", "ARCHIVES" or "PREVIOUS POSTS" fer the various RECIPEE'S....
Y'all Remember..."NEVER TRUST A SKINNY COOK".
Now THIS is Cookie's kinda "Boat"! The BEST kind of MARINE is a SUBMARINE!
Well Sir, we'll begin with this "Oldie But Goodie" thats timely fer this here time of the year....
_________________________________
Now Sir, fer those of you who are gonna be a fly'n this Christmas Season, here's sumthin to ponder....
____________________________________
A Christmas Story for people having a bad day:
When four of Santa's elves got sick, the trainee elves did not produce toys as fast as the regular ones, and Santa began to feel the Pre-Christmas pressure.
Then Mrs Claus told Santa her Mother was coming to visit, which stressed Santa even more.
When he went to harness the reindeer, he found that three of them were about to give birth and two others had jumped the fence and were out, Heaven knows where.
Then when he began to load the sleigh, one of the floorboards cracked, the toy bag fell to the ground and all the toys were scattered.
Frustrated, Santa went in the house for a cup of apple cider and a shot of rum. When he went to the cupboard, he discovered the elves had drank all the cider and hidden the liquor. In his frustration, he accidentally dropped the cider jug, and it broke into hundreds of little glass pieces all over the kitchen floor. He went to get the broom and found the mice had eaten all the straw off the end of the broom.
Just then the doorbell rang, and irritated, Santa marched to the door, yanked it open, and there stood a little angel with a great big Christmas tree. The angel said very cheerfully, 'Merry Christmas, Santa. Isn't this a lovely day? I have a beautiful tree for you. Where would you like me to stick it?'
And so boys & girls thats how the tradition of the little angel on top of the Christmas tree began. ____________________________________
One of my regular women-folk readers sent me this, "In My Next Life". I think my wife already made this decision....
____________________________________
The big bad Wolf said "I'll huff and I'll puff and I'll blow your house down."
The little piggy said "Screw off or I'll sneeze on you..." __________________________________
If'n Y'all ever owned and loved a dog, you'll enjoy this picture story
My good amiga, Sue Gertson frum down Texas way sent me this great photo story which no descriptive words are necessary.
Ifn y'all noted, everyone is wearing coats and warm garb so it had t'be pretty damn cold there. ______________________________________
Now Sir, on the other side of the coin we have this story. Although this did have a happy ending its the original event that boggles the mind and just piss's ya off....
8 Year old boy suspended from school for drawing a picture of Jesus.
A school superintendent in Taunton, MA has been ordered to apologize to the family of an 8-year-old boy who was sent home from school and ordered to undergo a psychiatric evaluation after drawing a picture of Jesus on a cross.
“The mayor is very upset and angry,” said Todd Castro, assistant to Mayor Charles Crowley told FOX News Radio. “He spoke with the superintendent this morning and is looking for her to make a public apology and a private apology to the family.”
According to the Taunton Daily Gazette, a boy at Maxham Elementary School was instructed to sketch something that reminded him of Christmas. The child had just returned from a visit to the National Shrine of Our Lady of La Salette – so he drew a stick-figure of Jesus on the cross.
In a small Texas town there was a "Nativity Scene" that showed great skill and talent had gone into creating it. One small feature bothered me. The Three wise men were wearing firemen's helmets.
Totally unable to come up with a reason or explanation, I left. At a "Quik Stop" on the edge of town, I asked the lady behind the counter about the helmets. She exploded into a rage, yelling at me,
"You darn Yankees never do read the Bible!" I assured her that I did, but simply couldn't recall anything about firemen in the Bible.
She jerked her Bible from behind the counter and ruffled thru some pages, And finally jabbed her finger at a passage. Sticking it in my face she said
"See, it says right here, 'The three wise men came from afar.'" _________________________________
I'm just an old Vietnam era Disabled Veteran Seabee, (and an old conventional Sub-sailer ) who's just a killin time...till time kills me. After retiring from many years in Law Enforcement as a Detective Sergeant, I now spends most a m'time a cookin fer feller sportsmen(and m' Wife) & writin a cook'n column fer a Fish & Game Magazine. I Hunt & Fish and don't care much fer them ACLUers. I live in a small northern NY town on Oneida Lake, NY, and like it that way. I can be as cantankerous as a Grizzly Bear with a tooth ache, or as cuddly as a Teddy Bear...mostly the former...