The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What�s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa�s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available �warts and all� for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you�re hoping to do with it. We may not be able to say �yes� to all requests, since we�re just figuring out whether this is a good idea, but everyone will be considered.

Total Users
257,093
..............................
Total Base Posts
6,085
..............................
Base Posts Today
0
..............................
Base Comments
8,618

Com-mu-ni-ty [kuh-myoo-ni-tee]-noun
1. a social, religious, occupational, or other group sharing common characteristics or interests and perceived or perceiving itself as distinct in some respect from the larger society within which it exists.

T-X-Base [tee-EX-beys]-noun
1. a gaming kuh-myoo-ni-tee created by TeamXbox.com for the people that live, breathe and die for the Xbox and Xbox 360.

2. see kik-azz.

Ladies and gentleman, boys and girls, and other life forms of all ages�are you ready for the most amazing gaming community the Earth has ever witnessed? (Um, that's a rhetorical question, so don't feel like you have to answer.)

It's called TXBase, and it couldn't be easier to use. Just sign up and follow our "How to Make a BasePage" instructions below. You'll be custom designing your very own BasePage and cohorting with friends, enemies and even industry professionals in no time.

What about functionality, you ask? Well, usually you get what you pay for, but in TXBase's case, it's free and awesome, which means you'll be able to express to the universe what moves you in the land of big black and medium-sized off-white boxes.

You'll be able to sound off and actually be heard; whether it's through your very own game reviews, editorials or general rants and raves. Your favorite and most-played games will be categorized, too, and plan on having your Gamerscore and achievements proudly-or perhaps shamefully-displayed for all to see. And we mustn't forget about our constantly-changing rank of the top BasePages in existence. Do you have what it takes to make this illustrious list?

Bottom line: Whether you're a hardcore gamer, a casual fan of the scene or one of those ladies or gents who is unsure of the difference between a cutscene and a combo�you need your very own BasePage, stat!

Want a sample of what a personalized TXBase page is all about? Check out our staff pages here.

Creating a BasePage on TXBase is as easy as following three steps:

1. Simply activate your BasePage. If you�re a member of the TeamXbox forums, your BasePage is already waiting for you and the only thing you need to do is to activate it!
   
   Don�t believe me? Then just go to http://base.teamxbox.com/tag/yourgamertag/ where �yourgamertag� is your Xbox Live gamertag. (tip: if �your gamertag� has spaces, use a dash for every space: so �your gamertag� would be �your-gamertag�).
   
   Obviously, this will work only if you are a registered member of the TeamXbox.com forums and have your Xbox Live gamertag associated to your TXB forum profile.
   
   

   Don�t have a TXB account?
   
   If you haven�t yet registered on TXB�s forums, not only should you reconsider some aspects of your life, but you also need to quickly go to this page to open an account on TeamXbox.com. There you will have to create a username and then associate your Xbox Live gamertag to it, if you have one.
   
   If you don�t have an Xbox Live account yet, you can still use TXBase, but in this special case, the URL for your TXBase page will simply use a number until you associate an Xbox Live gamertag to your profile.

   
   
   
2. You need to tell the world about your new BasePage, but if you have no friends, then how are you going to spread the word? Make sure you Add Friends to your profile and we can assure you that, in the blink of an eye, the entire Internet will know about your new evil lair.
   
   
   
3. The third and most important step is to start blogging, so you can tell the entire community what you are up to, what you think about the hottest topics, gaming and non-gaming related.
   
   In this beta version of TXBase, you�ll have your own blog space with the ability to insert images and links within your text. In addition, your BasePage will include:
   
   
   • User Info: Display your TXBase/TeamXbox.com Forum info.
   • Xbox Live Gamertag: Display your Xbox Live Gamertag.
   • Add / Show Friends: You can Add Friends from the TeamXbox community and they�ll be displayed here.
   • Most Played Xbox 360 Games: You can show your Most Played Xbox 360 games, your Gamerscore, the achievements earned for each game and the last time you played them.
   • Top Gamertags @ TeamXbox.com: You can also see the top Gamerscores from the TeamXbox.com community.
   • Top Achievements @ TeamXbox.com: You can also see who from our community has the largest number of achievements.

TXBase is a new place for the TXB community, and that means everyone starts at zero when it comes to blog entries count. Hurry up�the counter is ticking!