The Internet Archive discovers and captures web pages through many different web crawls.
At any given time several distinct crawls are running, some for months, and some every day or longer.
View the web archive through the Wayback Machine.
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.
What’s in the data set:
Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069
The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.
However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.
We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.
If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.
TIMESTAMPS
The Wayback Machine - https://web.archive.org/web/20110426041305/http://titanquestvault.ign.com/
Need an answer to your gaming question? Good! Head on over and check out IGN Answers for all your gaming Q&A; needs. Ask a game related question and get an answer quick.
We are excited in announcing the launching of our latest site, Halo Ring, for the mega-series by Bungie. So please take a moment to drop by and see what is going on over at the Ring. While there be sure to check out the Halo Ring Wiki section for all your needs and maybe add a bit of your knowledge while you are there. There will also be extensive beta coverage with guides, screenshots and editorials. Time to suit up!
We are excited in announcing the launching of our latest site LittleBigPlanet Factory for the ultra-fun series by Sony. So please take a moment to drop by and see what is going on over at the Factory. While there, be sure to check out the LittleBigPlanet Wiki section for all your needs and maybe add a bit of your knowledge while you are there!
A big hello to all fans of Titan Quest, Immortal Throne, and the once great IronLore!
For those who haven't been over to the TitanQuest.net forums, I wanted to let you all know that former Lead Designer of IronLore has moved on along with other former employees to form Crate Entertainment.
Their larger scale project Black Legion is awaiting funding, but in the mean time they are developing a smaller scale (probably limited to online distribution) game using the Titan Quest engine named Grim Dawn. This game is pre pre pre development but I think the screenshots are looking quite promising, you can head over to the game's website here where you can see screenshots as well as check the forums for game updates.
Also available is a contribution style preorder for those who are truly die hard. Please read the full text on what this means before you do it. :)
Hello fellow Titan Quest fans! I know it has been far too long since there's been updates to this page for TQ related tools so here goes nothing!
For those out of the loop, the latest TQ Vault (currently managed by EJFudd is now up version 2.3.0.3!! It has been updated to include things like caravan modification, and the ability to auto-update using my updater! I've updated the 'Editors and Tools' section to reflect the new version, or you can follow the official thread located here!
As many of you know the TQ Defiler(.NET) has been long overdue for an update as well, a new release has been put up tonight (1.1.2) including fixes and features that are new or were previously disabled during the 6.0 to .Net transition including gender and tunic color changing. You can find this under the 'Editors and Tools' as well or via the official website for download located here.
Hope everyone is still enjoying Titan Quest, the game still shines even today. :)
IGN, GameSpy and TeamXbox have rallied the entertainment industry to aid Haiti in its time of need. Today, IGN is broadcasting a 12-hour webathon, beginning at 9am PST and ending at 9pm.
The people of Haiti need your help. You can donate directly to our chosen charity, Habitat for Humanity, or you can jump to the IGN for Haiti eBay page to bid on some amazing auction items. Some notable items include a Life-Size Replica Bayonetta Gun, signed BioShock 2 Splicer Mask, Half-Life Guitar, and over 250 other kick ass items. All proceeds from the auction go directly to Habitat for Humanity to aid the effort in rebuilding Haiti.
Over the course of the 12-hour webathon, IGN editors will be playing a non-stop series of online Call of Duty: Modern Warfare 2 matches. IGN employees have pledged money for each level earned during our play session. You can find and play with us online all day today. Just look for the "IGN Xbox" gamertag. Help us rack up levels!
Looking to find the hard details for one of the campaign quests in Aion? Or maybe you are seeking the lowdown on your level 50 Archon Tribunus's Sword? Look no further than IGN's Aion Tower website for all your game database needs. The site has just launched a super extensive game database for the MMO title, Aion.
Saying it's extensive might be an understatement. The site goes beyond listing everything you can find in the game. These include: items, recipes, quests, zones, NPCs, skills, maps, and everything else. It goes beyond that by providing 3D viewers, Grind Planer, Stigma Planner, Item comparisons, browser plugins, and so much more. This database is impressive.
We are extremely thrilled to announce the launching of our latest site, Dragonica Online World, for the THQIce's F2P MMO title "Dragonica Online". So take a moment to drop by and see what is going on with the game and its community. While there, be sure to check out the Dragonica Wiki, dig up that elusive bit of info or leave your own tips for other players!
We are extremely thrilled to announce the launching of our latest site, IGN Music Hub, for the upcoming Activision's hugely successful "Hero" games with special attention on the upcoming Guitar Hero 5, DJ Hero and Band Hero. So take a moment to drop by and see what is going on with the up coming games. While there, be sure to check out the Hero Wiki section for all your needs and maybe add a bit of your knowledge while you are there!
We are extremely thrilled to announce the launching of our latest site, Aion Tower, for the upcoming NC Soft's MMO title "Aion". So take a moment to drop by and see what is going on with the up coming game and check out the beta if you can. While there, be sure to check out the Aion Wiki section for all your needs and maybe add a bit of your knowledge while you are there!
Another day, another Ding!, get your MMO Podcast fix with Episode 8 of Ding! This week Ocura, Nextweek and Jube, along with guest GaMeRz talk about Bio Breaks, Aion, World of Warcraft, CrimeCraft, Fallen Earth, Jumpgate Evolution, Champions Online, Final Fantasy XIV, D&D; Online, Age of Conan and more! Download and listen to Ding! Episode 8.
It was a busy podcast this week for Ding! Episode #7, chock full of people with opinions! In addition to the usual cast of characters, Nextweek joined us again, along with special guest "Skunkeen", Community Rep for Neverwinter Nights, Modder, Dungeon Master, Podcaster, and overall freaky dude who needs a change of pants. On the agenda this week:
I think with this final bit 'o E3 love, we can safely say that the event coverage is over? Maybe. Anyway, Peter "Nextweek" Eykemans made his way to E3 2009, where he managed to preview a number of games, talk to a few developers, and film the winner for IGN's "Editor for a Day" contest! Below you will find a list of previews and a final wrap-up from the event.
You've seen the trailers, viewed the screenshots, and if you were lucky enough then you may have even seen some of your favourite upcoming games live and in person at E3 2009. Now it's time to vote on what rocked your world (or didn't rock your world!) in the Voodoo Extreme E3 2009 Reader's Choice Awards. Get in there and support your faves, vote now!
Welcome to Episode 5 of Ding! An IGN MMO Podcast, E3 2009 edition. This week we had our E3 correspondent, "Nextweek", on hand to talk us through some of the MMO goodness that he saw while at the event. Titles discussed included:
Despite a slow week of MMO related news, the Ding! must go on! Check out the Ding! Episode 4 agenda below, and give the podcast a download. Next week we'll be doing an E3 Edition, and we'll have someone at the event to bring us all of the MMO scoops!
