
"Gentlemen, we have officially run out of everyone's money....who wants cake?"
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

"The more I learn about the universe... the less convinced I am that there's any sort of benevolent force that has anything to do with it, at all."-----------Neil deGrasse Tyson
As the 'Villain and others have pointed out, the collapse in value of all assets other than cash has crushed the net worth of the Baby Boomers. It has also had a disproportionate impact on the rich and near rich.The number of American households with a net worth of $1 million or more, excluding the value of their primary residence, fell 27% to 6.7 million in 2008 from an all-time high of 9.2 million the year before, according to a report from market research firm Spectrem Group.
"America has a lot fewer millionaires than when this economic crisis began," said George Walper, president of Spectrem Group, in a written statement.
But don't weep only for the 2.5 million fewer millionaires. The report, which is based on surveys of 3,000 affluent households, also showed the number of both multi-millionaires and aspiring millionaires plummeted last year.
Affluent households, defined as those with a net worth of $500,000 or more, declined 28% to 11.3 million from 15.7 million.
And, of course, it has only gotten worse since the beginning of the year.
The subsidized insurance program at the heart of the state's healthcare initiative is expected to roughly double in size and expense over the next three years - an unexpected level of growth that could cost state taxpayers hundreds of millions of dollars or force the state to scale back its ambitions.


Give the Gift of Stuff
Dear Barry:
I've had a bit of a bad luck patch over the last month (losing my job, watching my 401k completely disintegrate, etc., etc.) and ended up relocating from a high rise in Lincoln Park to a new neighborhood along the Fullerton underpass on the Kennedy Expressway. I was a bit worried about the move at first, but my new neighbors have been great. In fact on move-in day they greeted me with a grocery cart "welcome wagon" containing some lovely and practical gifts like cans of Sterno, cardboard, fortified wine, and a hypo-allergenic harmonica. I would like to show my appreciation with thoughtful "thank you" gifts. Can you recommend something nice that won't break my budget ($3.00 total for 6 gifts)? Please help!
Barbara in Chicago
Dear Barbara:
With my busy schedule of entertaining foreign dignitaries and celebrities at the White House, I know how important a well chosen gift can be. Two weeks ago, for example, we received a visit from British Prime Minister Gordon Brown. The Prime Minister brought a few housewarming gag gifts including a pen set made from a boat, a framed paper thing from another boat, and some old books by Churchill (not Ward, but that English guy). Obviously we wanted to return the nice gesture so I sent my interns out on a scavenger hunt for an appropriate present. They couldn't find anything in the West Wing, but luckily Costco was open and was running a 25-for-the-price-of-10 clearance sale in the DVD department. You should have seen Mr. Brown light up when he opened that sack of classic titles like "Wizard of Oz" and "Baby Geniuses 2." I like to think those DVDs helped cement our Anglo-American "special relationship" even if, as he mentioned to me, they probably wouldn't work in his European player. Thinking quickly, I told the PM I would send him an American DVD player as soon as I earned enough cash-back points on my Costco card. Crisis averted, but that episode taught me a valuable lesson: always keep a stock of gifts handy in case some foreign poobah or supreme religious figure or failing industry leader pops by for coffee. As a result, I make sure the Oval Office closet is filled with pre-wrapped Sham-Wows and Snuggle blankets and trillion dollar bailout packages for whatever gift emergency might arise......

