The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
TIMESTAMPS
The Wayback Machine - https://web.archive.org/web/20110429221500/http://cliffscrib.blogspot.com/
I’m not going to write another emotional tribute to a family today although the one I lost definitely warrants that. As I sit here outside after doing some chores I realized just how little time there is to really reflect on anything good or bad for too long. The lost one of my favorite people in the whole world it just so happened to come at a time when I am probably the busiest professionally and personally. It’s no one’s fault and I am not complaining. It’s just one of those things that make life so complicated.
On one hand we have all these events and circumstances that affect us and we need to deal with them as well as we can. On the other hand things are constantly moving and life keeps going so much that I don’t know how much you truly get to reconcile your emotions. I wish everyone in the country had a year’s worth of time to just make everything stop so you could take a month off from doing things without going into financial ruin and your life being destroyed because you couldn’t numb yourself enough to act like everything was okay. There would be less people holding in all that built up frustration and animosity towards other people that usually comes spilling out at the wrong time and causes even more issues. I would love time like that. The only problem would be that after Katrina I would have used mine up already.
Since we don’t have the ability to do anything like that the only other option is the balancing act between reminiscing about the past and dealing with lost and at the same time moving forward and building blocks for the future. So while I sit outside and watch the kids of the next generation of my family play in the water I want to say rest in peace to Anna Thomas. I’ll miss her laugh most of all. A lot of things have changed for my family but we are still okay because of the foundation.
I hope everyone has a good weekend and if you have friends and family you care about spend as much time with them as possible. You never know when things may change and you’ll need those memories to get you through those rough days. That’s how I get through them.
You should always strive to learn new things about yourself. You never get too old for that. This week I learned three new things about myself.
1.Although everything I planted a few weeks ago may die soon, I love gardening.
2.I went without my phone for about three hours today and realized I am addicted to constant information and technology.
3.I never really cared about him before but now I really don’t like Donald Trump.
It’s not even the fact that he thinks Barack Obama isn’t American or he used the term “the blacks”. It’s the fact that he’s getting all this attention. If I were a smarter person I could put the words together to explain how Trump’s sudden political popularity is the perfect example of racism, sexism, and class warfare. To put it in simple terms, only a rich white male with access to media could be taken this serious despite his past and lack of qualifications for a job like president. If Trump can get this much attention with his attitude and personality then I think Jay Z should run in 2016. By that time he’ll probably have more money than Donald Trump does now plus he and his wife has always had a good relationship with the whites.
I was surprised that the Republicans actually had a real vote on Congressman Ryan’s budget. I thought they were just starting off with the extreme and hoping that the Democrats would counter with something less drastic but still containing cuts that they could accept. My thing is that if we are in a crisis why everyone over 55 gets to keep their health benefits the way they are now. I think Congressman Ryan and his friends know older people vote and when older people start preaching about America spending too much and having to cut back they are not talking about anything having to do with them.
There’s no way the president could sign off on this plan but I could only imagine how the people from 50-54 years old would feel. You’ve been marking off the days on the calendar for when you can finally stop paying for that private insurance and get on Medicare and now it’s about to be taken away. I don’t mean to be disrespectful to any older people. I was raised to respect my elders. With that being said, it’s funny that people in their 50’s and 60’s made most of the decisions that got us in this situation to begin with and now they don’t want younger people to give up their safety net.
I want to tell David Simon and the folks connected with Treme that even though the mayor went ahead and demolished the block of blighted houses featured on your promotion pictures that you shouldn’t worry. We have dozens of blocks like that. You can choose a new one for each individual episode if you want to.
If the state of Michigan wanted to layoff 5466 teachers in Detroit and take over the city’s school system they should have waited until a flood and all the citizens are refugees before they did it. It’s easier to do those kinds of things when the people aren’t around to say anything.
The NBA playoffs are starting and the Hornets messed around and ended up playing the Lakers in the first round. I am torn because I can’t openly pull for another team going against my city but I have been a Lakers fan since I was a kid and first saw Magic Johnson and James Worthy in action. I made up my mind that I will just stay neutral and enjoy the games. My official prediction for the finals is Heat vs. Lakers with Miami winning in six games. I just can’t see anyone beating Wade and Lebron on the same team four out of seven games no matter who else is on their team. If Rick in Gentilly is reading this I want him to know that I haven’t forgot about our six pack bet from last year. I’m willing to go double or nothing.
I’m glad the government didn’t shut down today and it’s only because of the military. I would hate to be out enjoying myself this weekend knowing that men and women in uniform we love so much but don’t pay that well may be struggling. If it wasn’t for that I may have enjoyed watching the country panic for a few days. Now that the deal is done we have to see how many jobs got swept up in those cuts. That’s the sad part about any cuts. You know someone is feeding their families because their job is connected to one of those programs on the chopping block. That’s why politicians look like heartless bastards to me when they talk about spending cuts like they found the money under some sofa cushions or something. I guess their job has less value because they don’t work for corporations like the politicians do.
Politicians like the use the phrase that the government has to tighten its belt the way average American families do when they are in a financial crisis. "Tightening our belt" is my second favorite saying right now after "do more with less". It’s true that when money gets a little tight families make changes to keep things going but that’s entirely different than a crisis. When families are in a crisis everything is on the table until the crisis is over.
In a crisis I have to do things that I normally wouldn’t think about doing and let go of my pride until I am doing better. I don’t believe in borrowing money from family and friends. I feel like if I don’t have the money to do something for myself then it doesn’t need to get done. I work with what I have. That’s how conservatives feel about taxes and government programs. I get that but we are supposed to be in a crisis. If I was really in a crisis I would call my parents and some friends even though I really didn’t want to. In most families parents would be pissed if they found out you were sleeping in your car because you didn’t want to ask them. If the government is in a crisis did we have to extend those tax cuts the way we did? Couldn’t we have let them expire and revisit the issue of cuts once we got back on our feet?
If we are in a crisis why wouldn’t everyone including the corporations be for closing the tax loopholes that would ring extra money in? That’s like someone having a few thousand dollars in their 401K they could access and to their house from getting foreclosed on and just leaving it there. Retirement is important but not being homeless now would be far more important.
Fighting two wars and then getting involved in Libya if we are in a crisis is like someone paying two car notes, losing their job and then buying a boat. If you knew a regular person that did something like that you would think he’s an idiot. If most American families handled financial crisis like the government is doing there would be even more homeless people than they are now which would be bad because we wouldn’t want to pay to help them.
The need to stop saying we are in a crisis and just explain what’s really going on. The rich and corporations want to keep all of the money they can and have the power to make more with no restrictions. The middle class can’t be taxed anymore without starting a revolution and the poor have no lobbyists. When you don’t have enough money coming in to pay for what you have and you are not willing to do anything to change that something has to go. That is the general principle going into these budget discussions. We’ll see how those average American families make out in the end especially if we lose Medicaid and the retirement age is raised to 71. I predict we will be doing a lot of belt tightening and bunch of other things too.
Time and circumstances has a way of changing your opinion about people. There was a time not too long ago when black people in my generation shook their heads at people like Al Sharpton and Jesse Jackson. They always seem to be raising hell for petty issues and playing the race card when we were trying to move past that. The things they were yelling and screaming about and that old school stuff like marching and having rallies couldn’t get anything done. They were just embarrassing everybody and playing self appointed black leader.
Then President Obama came along. He’s elegant, always talks in a distinguished tone, and he always sounds like he knows what he’s talking about. As far as image and conduct goes he’s the black community’s dream. The closer he got to the White House, the more we wanted all the bootleg leaders to shut up and get out of the way. Poor Tavis Smiley has been trying to organize people for years. He said one critical thing about Obama and we sent him to an island somewhere. President Obama was the symbol that we didn’t need that kind of illegitimate, sensationalized, self appointed spokesman anymore. We were trying to get to the mainstream and they never give those kinds of people much weight when it comes to public opinion.
That was all fine and dandy until after the inauguration. I should have known something was up when Sarah Palin got as far as she did even after admitting she doesn’t even read magazines or newspapers. We got Joe the Plumber making speeches and appearances on behalf of a political party. Glenn Beck’s crazy ass got to go on the highest rated news network every night and say some of the craziest things about the president and all kinds of other people. He got so much attention and credibility that he actually had a rally in DC and thousands showed up. As crazy as people think he might be you have to admit that he had a lot of impact for two solid years. Bill O’Reilly is less over the top but his rhetoric is just as crazy most of the time and the president gave him an interview on Super Bowl Sunday. You can’t get more legitimate than that. Now the final straw is here as Donald Trump is questioning the president’s citizenship and thinking of running for office. I saw a poll this morning on Morning Joe that made him the favorite among Tea Party voters. He was at 20% and Sarah Palin was only at 12%. If she decides to run for president, what kind of foolishness is she going to have to say or do to compete with Trump and win those people back? We are headed downhill fast.
I want to apologize to Reverend Al, Jesse Jackson and anyone else who got caught up in the 2008 election fever. Apparently we had no idea just how much of the general population was willing to embrace lunacy. If Donald Trump can question the citizenship of the president and have anyone outside of his wife and children give thought to him running the country then Reverend Sharpton and people like him can do and say what they want to do. We may not always agree with their style but at least most of the time they have a legitimate complaint.
I just got up from one of the greatest naps of all time thanks to a fish plate and potato salad. It was a lazy day for me. I am probably the only person in the city that saw such a beautiful day outside today and still didn’t get up to do anything. Sometimes you just have to shut it down and recharge your batteries. I’ll get out and do something tomorrow to fulfill my good weather obligations. I did go out today and check out my week old garden. Children have a powerful influence because going outside last Sunday and planting seeds was definitely not on my agenda but I did it. I know nothing about gardening at all. I was online for an hour before I went outside just to make sure I had at least the basic instructions right. I have broccoli, tomatoes, corn, and sunflowers because they are pretty. I don’t really need to see any full vegetables because they will probably become food for all the hood wildlife like the possums and raccoons. If I just get a few leaves to come out of the ground it’s a win for me. I’ll keep you posted.
Today is my mom’s birthday. I actually had a chance to see her and tell her in person this year because she was here the last few days getting her New Orleans fix in. I think my family’s getting homesick. Usually for my mom’s birthday I try to come up with a poem or something nice like that because sometimes she reads this blog. My mom is not easily impressed and has only given me positive reviews for one of my tributes. Since I am being lazy anyway I might as well re-post the poem she liked. I’m using that old school Katrina survivor picture because her and my dad has lost so much weight she looks just like this now.
For all the times you had to come outside in your pink robe when I stayed out too long.
For the days we danced in the kitchen.
For giving me your eyes even though they don't open that wide like my daddy.
For the time I forgot my mouthpiece for the football game and you came on the sideline to bring me one.
For anytime you thought your kids were the best at anything even if everyone else thought differently.
For saying No and not caring what the rest of the mamas in the hood did.
For knowing how to cook the cabbage until it’s brown and sweet.
For knowing I like my gumbo with lots of File`.
For knowing that something is wrong with me even without me saying anything.
For every time I had a problem with a company or situation and thought to myself
“mama gonna handle this for me"... and you did
For showing me how a wife is supposed to be loyal.
For helping me keep it together to keep everybody else together after Katrina.
For teaching me to love Aretha Franklin.
For 1,275 plays of the Solomon Burke Soul Alive album.
For being willing to quit a job before you missed a school play or awards show for your kids.
For always fighting back.
For being protective and supportive.
For being my friend.
For having as much to do with the man I am as daddy does.
For being the first woman I ever fell in love with.
For already knowing that I cried when I wrote this.
Today former NOPD officer David Warren was sentenced to 25 years for killing Henry Glover after Hurricane Katrina. Former officer Gregory McRae was given 17 years for setting Mr. Glover’s body on fire after he was killed. I really don’t know how to feel about the sentencing because I always think things would be better for everyone if Henry Glover were still here with his family. Seeing people get long prison sentences really doesn’t do anything for me no matter how awful I think their deed was. Nevertheless this kind of sentence was necessary in a city that is trying to turn the corner on violence and crime. I don’t think the police department can reduce the murder rate on their own. The community has to take the lead in that but the police have to be a big part of it and it’s hard to come to terms with some of the things certain officers have done and cooperate with them. We can’t expect to clean up our neighborhoods with hardcore criminals roaming the streets. We also can’t expect to move the city forward when officers don’t value the life of every citizen. From my perspective today was a positive day.
There’s always going to be extremes on any side of an issue. There are some people in the community including some officers that think these two officers got a raw deal. They will point to the fact everyone was stressed and under pressure during that time and under normal circumstances these men would have never just killed someone and set their body on fire. I think if the stress and pressure was that bad they could have always just quit their job and evacuated. They would be in better shape than they are now. We can’t make excuses for what they did.
The other side is the folks that think both men are getting off easy. They wanted to see double the amount of time handed down. I saw Henry Glover’s aunt saying she expected David Warren to get life. I can understand how the family and others feel. If it was my family I would want a longer sentence too. A longer sentence isn’t going to bring Henry Glover back to life so there’s no such thing as real justice. I see a 25 year sentence and I think to myself that they almost got away with it totally if it wasn’t for news stories about it a few years after the storm. Before that Henry Glover was well on his way to being another random casualty of Katrina’s chaos. I know the family wanted more time but I am glad they at least get some closure. Besides, I don’t think spending 20 years in a federal prison as an ex cop who shot a black man and set his body on fire is no country club visit by a long shot. Both men will have more than enough time to think about what they did and pay for it.
Hopefully soon we can start to put this one behind us and wait for the Danziger seven to get their day in court. If David Warren got 25 years for shooting one black man, it will be interesting to see what happens to the guys that shot a man in the back and tried to take out an entire family. It seems like we never run out of things to deal with down here.
It’s difficult to maintain a positive mood in such negative times. Yesterday afternoon I was headed back to work and drove right up on a fresh murder scene. I was blown away because it was so early in the day at busy area and across the street from a school. It was a sad situation. I had just left a meeting full of positive people and watched a young brother who I admire receive an award. When I did get back to work I found out another gentleman who I admire had a new and exciting position at City Hall. In one hour the two positive things were balanced by the worst possible thing. Such is life for a man like me in the city.
I know I am not alone in feeling the way I do. It’s like every day we are fighting an image problem. It feels we are being attacked by the cats who display their ignorance for the entire world to see without any shame and all the people who use their actions to lump us all together. I don’t think we have the time or the resources to stop either one of those things from happening so the only thing we can do is make sure that we keep one another afloat. Whether it’s the college educated professional or one of the delivery men at my office building, my brothers need to know that they have value.
I try my best to give my brothers encouragement. When I am around a man that is trying to do something positive I let them know that they are good men. I tell them to keep moving forward. I try to explain all the things I did wrong and the bad choices I made so they won’t make them. There are a lot of ladies out here raising boys on their own. When I meet their sons and their heads are on straight I let their moms know that she is raising a good young man, to keep up the good work and call me if he’s going through something that requires a man’s perspective to help him with. There are plenty of impressive young men in our community but they don't get enough attention because our failures make better news stories.
Mutual admiration for one another is important. There are some things that should be routine like spending time with your kids and getting your education that should be so routine that men shouldn’t expect or receive any praise for. I would love to be at that stage in our community. I think one of the main issues we have is that we don't do enough to recognize the good things men do but the guys that do and say the wrong things get so much attention that it became popular and cool. Now we just keep trying to legitimize destructive behavior and it's not working. Since each day is a constant barrage of negative images and subliminal messages suggesting how cool it is to be ignorant, patting one another on the back for encouragement seems perfectly fine to me. Appreciation has a lot of value when you are trying to kill a destructive culture.
If you know a good dude shake his hand and let him know.
All Americans should have an opinion about war and military action. There are people in our armed forces from all walks of life and from every section of the country so it affects everybody. You don’t have to share it with anyone but I think it’s one of those topics everyone should think about. I don’t have a problem sharing my views. I think that if it’s a matter of national security then military action is a necessary evil and we need to do it. The problem in recent years is since information and truth has gotten so fuzzy it’s hard for me to determine what’s necessary and what’s not.
I don’t know if Libya was necessary. It seemed to me like they were having a civil war and some bad things were happening during the fighting. It’s not like Egypt where the majority of the country including the military seemed to put the people over leadership so army didn’t open fire on their own and create mass carnage. Libya seems to be a bit more complicated in that a lot of people still support Moammar Gadhafi. I could be wrong but that’s the way it looks to me. If I am wrong about the people I don’t think I am wrong about the military because they really shut the rebels down.
Now America is part of another coalition and is bombing Libya. The president says there will be no ground troops and I hope it stays that way. I don’t think it was a good idea to get involved in that country. I have two main issues. The first one is that Saddam Hussein was this evil dictator who harmed his own people like Gadhafi is doing right now. We removed Saddam from Iraq but the people of that country were so tribal and divided by religious sect that it took less than a year to whip Saddam’s army and then almost another ten years to try and get the people we freed not to kill us. We might be spending money to help Libya five years from now.
The second thing is whenever it’s a country that we have a vested interest in changing we always pull out the Protecting Innocent People card instead of just saying that we want the leader removed and get it over with. I don’t want to see innocent people get killed no matter what country they are in but innocent people have been getting killed in fighting all over the place especially in areas like Sudan and The Congo and we haven’t fired anything at anybody. It makes that argument look like we have no real reason to be doing what we are doing because it’s not consistent. The other thing part of that is how you know when those people are no longer in danger. If we stop the bombing tomorrow is Gadhafi just going to let bygones be bygones in those areas where the rebellion started? He’s not going to let it go so we can’t leave him in power and that means we are going to have to be involved in that country to make sure things go well. We might not have troops there but we will have to do something.
Regardless if you are for or against getting involved in Libya we should all agree that we will have to pay for something to keep this from blowing up in our faces. When it happens I am going to find it really sad that we got involved and financially invested in Libya’s business when we are supposed to be so financially strapped as a country that teachers are made to seem evil just for wanting good benefits.
Tonight is one of the best nights in a sports junkie’s life. It’s NCAA tournament time. Normally I would be watching CBS and letting Greg Gumble switch me to the game that is the most interesting at the time. Now I have to keep turning to all these different channels with no guarantee that anything exciting is going on. It doesn’t matter. I got beer. Everything is done for tomorrow so I’m going to sit here and watch Gonzaga and St. Johns.
There are a lot of things going on in the world. I’ve been watching Japan and hoping they can hold back most of the radiation. I don’t want to say anything misinformed or ignorant that disrespects the plight of the Japanese but I do have two thoughts about the nuclear problem. The first thing is that I don’t know how reactors work but I have to believe there’s a way to back up the back up system to the back up system before you start operating these things around people. I know that was a powerful earthquake with a tsunami to follow but this just seems ridiculous.
The second thing is that everyone who works in those nuclear sites is underpaid. I am sure they are making a lot of money right now and they are still underpaid.
”The patterns of policing in New Orleans are biased against several demographic groups, including black residents, people who don't speak English fluently, gay and transgendered people and women.”
Given the demographics of the city, this quote means that the NOPD is fair and balanced to about 500 people. I don’t have a problem with Chief Serpas personally. I just think he has the same weakness that Chief Riley had before him. There are too many personal allegiances to officers that need to be fired. It doesn’t make either man corrupt. It just makes them products of the culture of the city like the rest of us. Chief Serpas may eventually clean up the force but he’s going to have to be willing to not be invited to a lot of parties and seafood boils. That’s more difficult to deal with in your hometown then most people will admit.
The last thing I want to mention is the story about the 14 year old kid at McDonald City Park Academy who was put in an equipment locker. He called his mom saying he was being locked in a cage. I don’t know this kid or his behavior but there is no reason to lock them behind a cage at school if that’s what happened. I don’t know what the alternative is besides suspending them but being behind anything that looks like a cage conditions them for the wrong thing. You don’t want them to get used to that. In the story it is noted that the mom of the kid quit her job just to be able to get him on track. I don’t know if that means she wanted to be home in the evenings to help him or he was getting in so much trouble that she couldn’t stay at work anyway because she had to leave. I know one thing for sure. When you go to school with the kind of kids that need to be removed and isolated from everybody else there’s no way your teacher is going to be able to focus on educating you. She’s going to spend the entire time trying to get them under control and little time showing you what you need to be successful.
When I was in 7th grade there were a few guys that I thought needed a cage, medication and shock treatment. My poor old 7th grade English teacher spent an hour and a half fighting and dodging things being thrown at him and the other 30 minutes trying to remember what he was supposed to be teaching us. I guess that’s why he just gave up and passed everybody who turned in the ten definitions for homework. When people start talking about low test scores and horrible graduation rates these are the kind of things that lead to that happening.
Governor Jindal is going to merge the University of New Orleans and Southern University of New Orleans. The governor hasn’t shown that he can be moved from his position much from rallies and emotional testimony so I don’t see him changing his mind. Regardless of how you feel about the governor you have to admit that he is a good politician and knows his state. He already knows that merging UNO and SUNO won’t do much financially in terms of saving the budget for higher education but it will help ease the pain when he raises tuition across the state. It’s the symbolism since both of those schools have New Orleans in their name. Having Jesse Jackson come in on SUNO’s behalf makes it even better. While everyone is feeling good about him sticking it to the folks in the city he’ll be making it more expensive for them to send their kids to school and it won’t get much coverage. I know it’s going to work out this way because even some of the students at UNO who have been interviewed about the merger on television have made comments like they are getting prisoners from Angola instead of other college students from right down the street.
The biggest thing to me in this entire process is how the supporters of SUNO are falling victim to the same weapon that has led the charge for every initiative implemented in the city the last few years. The statistics are not on their side. Whenever someone wants to make a change or take something out of our hands like we can’t do for ourselves they throw out a bunch of negative statistics and we have nothing to counter that with but emotion. Emotion doesn’t have much value when you don’t have revenue, don’t show up to vote and have lost political influence. It kind of makes people wonder why we fight so hard to maintain a status quo that hasn’t done much for us. It doesn’t matter if it’s crime statistics, standardized test scores, graduation rates or anything else you can think of; it appears that we didn’t really do a good job at taking care of our people and institutions.
A lot of people who go to SUNO never get the chance to graduate in four straight years because life gets in the way. They get a better job or change schedules. Something happens so they take off a semester or two and then go back. I know a few people who started making money without the degree and didn’t go back at all. You don’t get extra credit for those situations when graduation rates are involved but the value in the school is that it’s a place where people from the community feel comfortable enough to turn to if they decide to try and better themselves. I think that’s the way the folks at the school see their mission even though I don‘t think they have done a good job at articulating that since the merger discussion began. You can’t put that into any statistical category either so I think that in this current climate if the governor goes with the plan that will allow those kind of students to still have a chance at improving their situation then we are going to have to settle for that. The numbers don’t support anything that would stop the governor’s plan.
