The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
TIMESTAMPS
The Wayback Machine - https://web.archive.org/web/20110429035427/http://www.neoneocon.com/
For those of you who might not be familiar with her, she was a young phenom in the 70s, a singer/songwriter with a uniquely bluesy style. Her life was not an easy one; a hydrocephalic daughter and a divorce took a huge toll on her career, and after the beloved daughter to whom she was dedicated died in 2007, one wonders if Snow lost some of the spark that had animated her own life.
But man, what a singer she was (the tape’s from 1989):
This acted as a signal to supporters—as if they needed urging—that criticism of Obama was to be interpreted as racist. Obama himself would stay serenely above the fray, the post-racial healer, and the attack dogs would attack.
Whether or not those playing the race card actually believed their own charges, or whether they just figured the approach would be strategically effective in both demonizing the opposition and silencing it through fear of being called racist, it all worked pretty well for quite a while. Early opponents Hillary Clinton and then John McCain were effectively neutered by it, the “first black president” Bill Clinton was enraged, and the whole ploy helped Obama ride the tide of hope and change to power.
By now, though it’s gotten pretty old. After a while the American people began to wonder whether any criticism could be mounted against Obama without triggering the old “racist” canard.
Of course there are still some racists in America, in both parties. But just as surely, the vast majority of the criticism of Obama comes from disagreement with his policies, well-earned distrust of his word, and dislike for his much-displayed narcissism. To notice those things and comment on them is to treat him as what he is: a human being to be judged on his merits or lack thereof, not a specially favored, criticism-exempt member of a minority group.
The birther controversy was a great gift to those crying “racism,” because it allowed them to revive the “racist” charges against those who doubted Obama’s American birth, or those who merely wanted him to document it (as others before him have had to do—the very white but Panama-Canal-born John McCain, for example, a fact that the anti-birthers conveniently ignore).
Today the NY Timesoffers its typically sneering and dismissive, mildly Orwellian take on the matter:
With sardonic resignation, President Obama, an eminently rational man, stared directly into political irrationality on Wednesday and released his birth certificate to history…It was particularly galling to us that it was in answer to a baseless attack with heavy racial undertones…the birther question was never really about citizenship; it was simply a proxy for those who never accepted the president’s legitimacy, for a toxic mix of reasons involving ideology, deep political anger and, most insidious of all, race. It was originally promulgated by fringe figures of the radical right, but mainstream Republican leaders allowed it to simmer to satisfy those who are inflamed by Mr. Obama’s presence in the White House.
Actually, it was originally promulgated by the Clinton camp, but why should the Times care about history or veracity in this particular matter when it cares so little about it in general? Besides, what is irrational about asking Obama to produce a long form birth certificate, and what is “eminently rational” about his failing to do so all these years when he could have scotched the controversy long ago?
Speaking of rationality—you may ask why I still pay any attention at all to the NY Times at this point. Well, it’s for a number of reasons, chief among them the fact that I still know a lot of people who consider it the final word on what’s happening in the world. Another is that when I was growing up the Times was the newspaper of record, and “all the news that’s fit to print” was something I actually believed.
What can I say; I was a child. Looking back, I don’t think the Times was ever what I was told it was or what I imagined it to be. But my feeling of betrayal and disillusionment is still there.
Racism is so hard to find these days that it must be embedded, hidden, covert, indirect—that is, “coded.”
So of course Donald Trump—who is many things, including a self-serving, narcissistic, publicity-hound—is of course a coded racist for questioning Barack Obama’s credentials and asking whether his entry into Columbia or Harvard Law was due to affirmative action.
Well, affirmative action was and is a reality. And Obama’s failure to reveal his transcripts feed the question and make it a valid one.
This accusation that Trump is a racist is actually sort of funny, though—that another coded racist remark of Trump’s was his request that Obama “gets off his basketball court or whatever he is doing at the time” and work on dealing with the high gasoline prices at the pump.
Well, Trump smoked it out: Obama’s long birth certificate finally sees the light of day.
Why did it take so long? Simple. Prior to this, Obama felt that secrecy was to his advantage. The controversy merely made his foes look bad. But something about the Trump accusations seems to have tipped the balance in another direction. Too many people were falling, if not into the birther camp, then into the “So, why doesn’t he just release it?” camp.
And so we have the long form at last, plus Obama’s characteristically condescending remarks to go with it:
We do not have time for this kind of silliness. We’ve got better stuff to do…Over the last two and half years, I have watched with amusement. I have been puzzled with the degree with which this thing just kept going.
Apparently, until now we did have time for such silliness, and no better stuff to do. And of course the fact that he wouldn’t release the form had absolutely nothing to do with the “degree to which this thing just kept going.”
Note, also, the use of several of Obama’s favorite words, “silly,” “stuff,” and “amusing.” I noticed a very similar pattern back in April of 2008, when I wrote (referencing these remarks of Obama’s):
Did you ever notice how often Obama uses the word “silly” to describe questions he doesn’t want to answer and issues he doesn’t want to address?
It’s an interesting ploy, a way to escape being pinned down while attempting to avoid charges of evasion. He could answer, of course. But he doesn’t want to waste his time on “silliness” because he, Obama, has a higher calling.
But this is not condescending. Not in the least.
Elitism? It’s a silly charge that “amuses” him.
I cannot in my lifetime remember any president, Democrat or Republican, good or bad or in-between, adopting this sort of patronizing, belittling tone towards his opponents (and just one Vice President, Spiro Agnew). From Obama, it’s standard operating procedure.
In fact, another thought has just occurred to me: perhaps his long delay in releasing the long form has been to set up just this moment, so he can rise above it and belittle the little people who so stupidly have provided for his amusement all this time.
[NOTE: Here’s a not-atypical response from the left, and an example of its “logic.” Weigel writes:
And the long-form certificate proves that the COLB was legitimate and answered all reasonable questions about Obama’s birth. If you were “just asking questions” about the COLB, you were not a truth-seeker. You were a fool.
Weigel is probably correct in saying, however, that “I’d expect the focus to shift now to Obama’s college records and writings.” I’m sure he thinks anyone who wants to see said records is also a fool. Of course, anyone who wanted to see McCain or Bush’s transcripts was probably a very reasonable and intelligent person. That’s just the way it goes.
In case you don’t remember who Weigel is, see this.]
[ADDENDUM: Andrew Sullivan has a moment of sanity on the subject. However, my guess is that Sullivan’s call that Obama should have released the long form long ago is based on Sullivan’s own demands that Sarah Palin release each and every document connected with the birth of Trig, in order to prove the child is hers.
At least he’s consistent, sort of.]
Posted by neo-neocon at 1:17 pm. Filed under: Obama
Ballet specializes in transformations. After all, it changes the human body from a lumbering earthbound thing into an ethereal yet steely and gravity-defying, lighter-than-air, transporter of the audience into realms of beauty and transcendence.
Yes, it can be corny, especially the old warhorse ballets such as “Swan Lake” and “Giselle,” featuring doomed maidens and supernatural spells—and in the case of the former a princess who is half-swan half-human.
But that’s part of the fun of it all—going past the schmaltz and the drama and creating something Other. The abominable film “Black Swan” traded at least partly on this mystique.
One of my favorite moments in dance occurs at the end of the second act of the ballet “Swan Lake.” I’ve seen it countless times. To those of you who don’t know the story, it sounds quite absurd, and it is: the prince has been out hunting, sees a swan he’s about to shoot, but before he does so the bird transforms before his eyes into a swan-maiden (whatever that is). They fall in love, and she explains that an evil magician has put her under a spell. After he plights his troth (which occurs in dance when a person touches his/her heart and then holds two fingers aloft; it’s a mime thing) the magician comes out and changes her back into a swan again and she flies off.
That latter part is the moment I’m talking about. On the stage, when it’s well-done, it never fails to give me a chill. You can see the moment the spell starts taking hold, then her struggles to fight against it, and then the instant when the spell totally overwhelms her and she is transformed into a flying bird.
Here are a number of versions of the same moment, all with somewhat different choreography. None of them, of course, have anything like the same magic and illusion they convey on the stage. But they’ll do:
This next one is the great Russian dancer Plisetskaya (see this):
And here, for variety, is the much more restrained British dancer, Margot Fonteyn:
Posted by neo-neocon at 1:56 pm. Filed under: Dance
Commenter “Daniel in Brookline” calls Trump a gadfly. It’s a word I’ve thought of often in connection with him and the Obama presidency, one that was originally used by Socrates to describe himself.
Here’s an explanation of what Socrates meant when he used the term [punctuation and spelling corrected]:
Socrates liken[ed] himself to a GADFLY (a horsefly). Just as a gadfly constantly agitates a horse, preventing it from becoming sluggish and going to sleep, so too Socrates, by moving through the city stirring up conversations in the marketplace, prevents the city from becoming sluggish and careless and intolerant.
It fits pretty well, although Trump’s most assuredly no Socrates. More and more, he also reminds me of a jester, although not one directly in the employ of the Obama court.
Why a jester? Well, he combs his hair funny. But mostly it’s because he fulfills this function of the traditional jester:
In Renaissance times, aristocratic households in Britain employed licensed fools or jesters, who sometimes dressed as other servants were dressed, but generally wore a motley (i.e. parti-coloured) coat, hood with ass’s (i.e. donkey) ears or a red-flannel coxcomb and bells. Regarded as pets or mascots, they served not simply to amuse but to criticise their master or mistress and their guests.
Jester/fools could say things no one else could say, ask questions no one else could ask, because they had little to lose and were given license to tweak. Trump has already voiced the unspeakable birther concerns, and now he gets into the very un-PC question of how Obama gained admittance to the two Ivies, Columbia and Harvard.
The insinuation is that Obama was either promoted by nefarious and unspecified helpers or was a beneficiary of affirmative action. Either suggestion could be scotched by Obama releasing college transcripts that show evidence of his stellar academic performance at Occidental (indicating he had the grades to get into Columbia) or Columbia (indicating he had the grades to get into Harvard Law School).
It’s really quite simple, but Obama has not done it, mainly because he’s never had to. People have always taken his brilliance for granted, as evidenced by his deep and sonorous voice uttering words mostly written by others.
I have no idea what Obama’s grades actually were, but I’d certainly be interested in knowing. It’s not that I think grades are the measure of all things, either; plenty of intelligent people don’t have very good ones. But Obama has always ridden on the idea that he’s an academic star, and it would be nice to know if he really was one.
It does seem to be clear, however, that once Obama got to Harvard Law he did pretty well, graduating magna cum laude, which according to Harvard means in the top 10% of the class (his Law Review presidency was an elective office, and therefore does not count). But that’s after the fact. When he graduated from Columbia it was a different story:
Obama graduated from Columbia University (to which he transferred after his first two years at Occidental College in California), with a degree in Political Science without honors, so had a GPA less than a 3.3.
Trump appears to have hired a team of investigators to look into this sort of thing (he certainly has the money to do so). He has said, “”I heard he was a terrible student, terrible…How does a bad student go to Columbia and then to Harvard?” He doesn’t offer evidence that Obama was a bad student, but the question of what sort of student Obama was is a valid one, and it’s something the MSM should have asked back in 2007, when Obama officially became a presidential candidate.
If you look at much of the commentary on Trump’s question, you will find the query treated as either a joke or a racist outrage or some combination of the two (see this, this, and this, for example). It shows just how untouchable Obama still is to many people.
Are they frightened of what such an inquiry might reveal? I don’t know, but my guess is that many of them truly do believe that Obama is a genius and of course his transcripts would reflect that. And that Donald Trump is a fool.
Or a Fool.
[ADDENDUM: More from Ace on the subject of Obama’s academic record. Read the whole thing. But one of the most interesting parts is the following:
Where I went to school, magna was reserved for the top 5%. When Obama was at Harvard, it was much, much easier to grab that accolade — only after he graduated did they institute a policy limiting magna to 10% and cum laude to 30%, which means, of course, before he graduated the honors were given out much more freely.
How freely? This freely:
“Under the old system, 76% of Harvard Law grads earned honors, the school said.”
Eric adds:
“So when Obama graduated in 1991, all he had to do was graduate in the top 76% of the class to get a cum laude honor. The article doesn’t specify, but I expect that the cutoff for Magna Cum Laude was 50%. Which means a C student could graduate with a Magna Cum Laude.”
I had always read that a magna degree from Harvard Law meant Obama was in the top 10% of his class, which would be an impressive achievement. This new information casts a great deal of doubt on that. Here’s the full article from 1999:
When members of Harvard Law School’s class of 1999 receive their prestigious degrees Thursday, 36% fewer graduates than last year will be awarded with honors, the school said Wednesday.
The relative scarcity of honors grads is the result of a new policy intended to boost the value of a cum laude degree and remove the stigma associated with not graduating with honors, Harvard Law spokesman Mike Chmura said.
Under a system implemented three years ago that first took effect with this year’s class, Harvard Law said it will limit magna cum laude degrees to the top 10% of the class. The next 30% will receive cum laude degrees.
The policy for summa cum laude, the highest honor, is unchanged. One summa degree was awarded this year, Harvard Law School said.
Under the old system, 76% of Harvard Law grads earned honors, the school said.
In the summer of 1978 I traveled to Europe. My first stop was London, a place I’d never visited before.
I was surprised at some things I found there. One was that in July the weather was really nice—sunny and clear and warm but not too warm. Another was that the bathrooms were dirty and the toilet paper strangely rough or slightly waxy. Still another was that I witnessed nearly constant altercations of a racial/ethnic nature in the subways and on the street.
But the most surprising thing to me at the time were the newspapers. It seemed that no serious newspapers remained; they were all tabloids. Partial nudity and screaming headlines about this or that sexual escapade of famous persons, and trivia and gossip galore—this was what now passed for news in that previously-staid previous-capital of the world.
The fact that I was surprised and even shocked seems rather quaint and almost amusing now, because that type of news coverage has become commonplace in the mainstream media here, too—perhaps minus the more salacious photos, but otherwise very similar (unless, of course, your name is John Edwards, in which case it’s limited to the Enquirer).
Journalistic standards were always far from perfect. Nevertheless, in my youth they used to be better. For example, the use of the anonymous source was rare rather than commonplace, unlike today. Gossip was published on the gossip page, and it mainly concerned Hollywood celebrities rather than rumors about politicians.
In the treatment of Sarah Palin, the press has reached a new high (or low, depending on how you look at it) in the elevation of gossip and malicious rumor-mongering couched as news, or at least newsworthy speculation. I have decided to call the phenomenon “Palinhate.”
The sheer scope and breadth and the relentless nature of the assault has no precedent in public life, as far as I can tell, and that includes anything launched against President Bush or his predecessor (see this). The attacks on Palin began almost immediately after her nomination, have continued unabated to this day, and have taken a huge toll on her reputation, which is the goal.
This rumination of mine today was prompted by this Politico piece by Kenneth P. Vogel. The impetus for Vogel’s article is a flurry of Palinhate books due to come out soon, just in time to attempt maximum damage for the 2012 election:
A number of forthcoming books promise to delve deeply into — and, they believe, give mainstream credibility to — some of the more salacious Palin rumors and conspiracy theories that have sprouted in the anti-Palin blogosphere and on supermarket tabloid stands but have mostly been rejected by the mainstream media.
“We’re at a tipping point, where her character and her lack of ethics will be revealed on the national stage,” asserted Sherry Whitstine, a 49-year-old grandmother who lives in Palin’s hometown of Wasilla, Alaska, and has infuriated her famous neighbors with blog posts and online comments accusing Palin of being unfaithful in her marriage and corrupt in her political career.
These books will most likely be based on either anonymous sources or identified disgruntled neighbors and former associates with a grudge. This is what passes for journalism today. Why is it so popular? Because very often it works, and very often it sells.
Posted by neo-neocon at 2:15 pm. Filed under: Press, Palin
Happy Easter to all my celebratory Christian readers, and to all those who just enjoy the holiday as well!
One year when my son was little, I spent the week prior to Easter blowing out eggs and dying them. Now that he’s grown and away, the eggs are packed away in boxes and stored in parts unknown. If I could get my hands on them I’d photograph them for you, because even all these years later they are beautiful, with dyes both subtle and unsubtle, interesting etched patterns and rainbow effects—definitely one of my finest crafts hours (to tell the truth, I didn’t have so many fine crafts hours, although there was also a gingerbread house we made that was stored in the attic and alas, eaten by small creatures–and not human ones, at that.)
Blown-out eggs are well worth the trouble, and why? Because they last. And nothing eats them. You only have to make them once, and you’re all set. They are a bit fragile, but not so very.
First, you’ll need to make a tiny pin hole on each end of the egg. A pin works well, or a wooden kitchen skewer or even the tip of a sharp knife. Gently work the tip of the pin/skewer/knife in a circular motion until a tiny hole appears. Repeat on the other side. Then insert the pin or skewer (the knife will be too big here) far enough into the egg to break the yolk. Use your mouth [blow] to expel the contents of the egg.
And here is a more complex–but perhaps better–way, for those obsessive-compulsives among us.
These aren’t mine, but they’ll have to do as substitute:
[NOTE: This is a repost from Easters past. But it still works for me.]
Posted by neo-neocon at 1:03 pm. Filed under: Uncategorized
Our friend Paul Krugman tackles the idea that the right is misrepresenting the facts when it says the rich pay a lot more in taxes than the poor:
The claim that only rich people pay taxes is a zombie lie — something that keeps coming back no matter how many times it’s killed by evidence…
Yes, high-income people pay the bulk of the federal income tax. But that’s not the only tax! And while the income tax is quite progressive, the payroll tax — the other major federal tax — isn’t; and state and local taxes are strongly regressive.
Krugman then offers a chart based on research by a group called the Citizens for Tax Justice (here’s a link to the more complete version of their charts and research).
I find this worth talking about because, despite the straw man argument with which Krugman begins his comment (”the claim that only rich people pay taxes”) and his later typical accusation that the right is purposely lying, he raises an interesting topic: how much should the rich contribute? How much do they actually contribute? And what is a fair share for rich and poor and everyone in between?
Of course, a book—or several—could be devoted to these questions. But let’s just deal with a post right now.
To start with, I noticed—by the title of the group offering the chart Krugman cited, “Citizens for Tax Justice”—it can hardly be considered an objective source (here’s the Board of Directors of its research arm, the Institute on Taxation and Economic Policy). But no matter; I don’t have competing figures from a truly objective source (are there any?), so let’s just work with what we do have (their figures, by the way, are based on a method explained here).
Krugman is absolutely correct that income tax is only part of the picture, and is progressive in the sense that income tax rates (not just amounts collected) rise with income. Capital gains taxes and estate taxes (and to a certain extent interest and dividends taxes) are similarly progressive, and are also borne disproportionately by the wealthy because the wealthy tend to be the ones with those assets. Then there is the payroll tax, which is arguably regressive (see this for the arguments pro and con), but is also at least theoretically not a regular tax but is instead supposed to go towards later entitlements such as social security and Medicare. Those are the major federal taxes involved.
State taxes fall mainly into the categories of property and sales taxes, and are not progressive. However, once again, if the rich own more property and buy more goods, they will pay more tax in terms of total amount, although not in terms of percentage of income or tax rate. Some states also have income tax and/or taxes on interest and dividends, which vary in their progressiveness from state to state.
I’ve often said that economics and finance are not my field of expertise, so I may be making some errors here in the details, but that’s my understanding (have fun correcting me in the comments section if you so desire). The point is that the situation is nothing if not complex, but the bottom line is that even Krugman’s charts, furnished by the left-leaning Citizens for Tax Justice, indicate that the tax code does result in the rich paying not only a great deal more money in taxes in the sense of total amount, but more in terms of percentage of income as well. Therefore the entire system is progressive.
Who are these rich? The charts don’t tell us, except in terms of quintiles of the population. If you look at the graphs, you will notice that the three lowest quintiles pay not only a smaller amount of taxes in general (not just income taxes), as one might expect, but a smaller percentage of income as well. In the fourth quintile that begins to change, and in the fifth quintile the percentage of income paid is considerably larger.
Krugman writes, “The overall system is barely progressive at all.” His point is obscure; the system is progressive overall, not just the income tax part of it. What’s more, he does not address the point the right makes, which is (a) that the rich contribute a huge percentage of tax revenues compared to their actual numbers, and (b) that the lower quintiles do include many people who pay no regular income tax (as opposed to payroll tax) and therefore have an incentive to raise income taxes for the rich only. Another point is that the left and many of these people would like there to be more outright income redistribution as a result.
One can either agree or disagree with that goal. One can also believe taxes should or should not be progressive. But right now our overall tax structure (not just the income tax) is already progressive, as even Krugman concedes. One of the differences between left and right is that the left thinks it should be even more so and the right does not. Here is a summary of the arguments pro and con; judge for yourself.
Perhaps the most interesting detail, however, is one omitted by Krugman’s charts, and that is what those top quintiles represent in terms of actual income. I’ve been unable to find a relevant chart for the year 2010, the one that corresponds to the chart Krugman used. But if you look at this chart for 2004, you will see that the fourth quintile starts at an income of $55,000 per household, the fifth quintile begins at $88,000, and the top 5% starts at $157,000. The figures may have gone up somewhat in the intervening years, but my guess is that they haven’t jumped substantially.
I don’t know about your definition of rich, but mine does not involve households earning $55,000 or $88,000. So we don’t seem to really be talking about rich people at all. We’re talking about basic, middle-class, working people—particularly married couples, who tend to be disproportionately represented in those top quintiles.
So forget the super-rich, those who have great tax accountants and shelters anyway, and can move abroad if they wish—how much more can the middle-class be tapped to support the federal government and entitlements? Isn’t that really the question here? Are these groups not already paying their fair share, and then some?
Previously a lifelong Democrat, born in New York and living in New England, surrounded by liberals on all sides, I've found myself slowly but surely leaving the fold and becoming that dread thing: a neocon. Read More >>